You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/10/28 22:50:03 UTC
[GitHub] [flink] afedulov opened a new pull request #17598: [WIP][FLINK-tbd][connectors][formats] Add FileSource support for reading CSV files.
afedulov opened a new pull request #17598:
URL: https://github.com/apache/flink/pull/17598
This PR adds a new implementation of `StreamFormat` called `DelimitedFormat`. The format allows breaking consumed data by a specified delimiter pattern and deserializing produced blocks of bytes with a supplied `DeserializationSchema`. This approach is then tested in the context of reading CSV files using `FileSource`.
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
Adds the ability to read CSV data in `FileSource`. The underlying approach is also generalizable to any data format that needs to be split by a custom delimiter pattern into byte blocks with their further deserialization via the `DeserealizationSchema` API.
## Brief change log
- New `DelimitedFormat` is added and tested for the purpose of reading CSV data into `RowData`.
## Verifying this change
This change added tests and can be verified as follows:
- Added integration tests for reading CSV data using new `DelimitedFormat` via `FileSource`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): ( no )
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( yes )
- The serializer: ( no )
- The runtime per-record code paths (performance sensitive): ( no )
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no )
- The S3 file system connector: ( no )
## Documentation
- Does this pull request introduce a new feature? (yes )
- If yes, how is the feature documented? ( JavaDocs )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e5445170476fae8081037143c62c36318677ad6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364)
* 0c1c04317c7fb3f4470cdf596756172105d25cda UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-1018648798
Thanks @fapaul, I addressed your comments, squashed the hotfix commits and rebased on master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
* f883156429a6510b06a05e58ddd77fb8e0554516 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
}, {
"hash" : "f51dee82fb528a7ae6f883c46be76b3e1b3af6c4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f51dee82fb528a7ae6f883c46be76b3e1b3af6c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183)
* f51dee82fb528a7ae6f883c46be76b3e1b3af6c4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
* f883156429a6510b06a05e58ddd77fb8e0554516 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r792676507
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void> converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(), null) : null;
+ while (true) {
+ try {
+ if (iterator.hasNext()) {
+ R nextElement = iterator.next();
+ return converter.convert(nextElement, null);
+ } else {
+ return null;
+ }
+ } catch (Throwable t) {
Review comment:
I decided not to reinvent the wheel and handled it exactly as this option is currently handled in the existing implementation to avoid surprises for the users that already rely on this functionality:
https://github.com/apache/flink/blob/16452fbb57c11418796912251152947601572106/flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvRowDataDeserializationSchema.java#L157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
* f883156429a6510b06a05e58ddd77fb8e0554516 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181)
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 29e6025b084eb1c850ab3901c757dcfcb442c67e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289)
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "6f95051fa8153ff7745b479e75e44e9b0ea496f0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6f95051fa8153ff7745b479e75e44e9b0ea496f0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 29e6025b084eb1c850ab3901c757dcfcb442c67e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289)
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
* 6f95051fa8153ff7745b479e75e44e9b0ea496f0 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770834164
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.unixTimestampToLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class CsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
+ public void testProjectPushDown() throws Exception {
Review comment:
Will work on that, but first a small clarification:
Do you mean testing the new `CsvBulkWriter` + `CsvReaderFormat` and not anything related to the ser/de schema-based (old) stuff? Another question - you point at the projection pushdown test, not sure if you meant it explicitly (new functionality?) or if it is a coincidence. Do we even have facilities for communicating pushdown from the streaming API?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770835661
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormat.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
Review comment:
Do you have a good inspirational example for these kinds of instructions in our codebase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
* 949b964bcb65db05a9c903a28f59f933300a7935 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r771428588
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
Review comment:
Our current `CsvBulkWriter` is pinned to be used in the Table/SQL context because it relies on the `RowDataToCsvFormatConverterContext`. It seems that we might need a different approach given that using it in the DataStream API is also a requirement.
https://github.com/apache/flink/pull/17598/files#diff-a32dfa08e3b14134b196779c88529037e79fd77d350de57d7738dcd4d295f9eaR81
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 744b699a82b4640652050eb75eecaf78b2f10c2c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r779607851
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/TableCsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.toLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class TableCsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
Review comment:
We will probably take over all testing utils for formats. The right way would be to convert the `JsonPlanTestBase` into a junit extension and get rid of the inheritance.
I can understand if you do not want to do it as part of this PR but eventually with the next format we have to do it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
* 949b964bcb65db05a9c903a28f59f933300a7935 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
slinkydeveloper commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770302212
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.unixTimestampToLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class CsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
+ public void testProjectPushDown() throws Exception {
+ List<String> data = Arrays.asList("1,1,hi", "2,1,hello", "3,2,hello world");
+ createSourceTable("MyTable", data, "a bigint", "b int not null", "c varchar");
+ File sinkPath = createSinkTable("MySink", "a bigint", "c varchar");
+
+ String jsonPlan = tableEnv.getJsonPlan("insert into MySink select a, c from MyTable");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("1,hi", "2,hello", "3,hello world"), sinkPath);
+ }
+
+ @Test
+ public void testReadingMetadata() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.smallData3()),
+ new String[] {"a int", "b bigint", "m varchar metadata"},
+ new HashMap<String, String>() {
+ {
+ put("readable-metadata", "m:STRING");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a bigint", "m varchar");
+
+ String jsonPlan = tableEnv.getJsonPlan("insert into MySink select a, m from MyTable");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("1,Hi", "2,Hello", "3,Hello world"), sinkPath);
+ }
+
+ @Test
+ public void testFilterPushDown() throws Exception {
+ List<String> data = Arrays.asList("1,1,hi", "2,1,hello", "3,2,hello world");
+ createSourceTable("MyTable", data, "a bigint", "b int not null", "c varchar");
+ File sinkPath = createSinkTable("MySink", "a bigint", "b int", "c varchar");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select * from MyTable where a > 1");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("2,1,hello", "3,2,hello world"), sinkPath);
+ }
+
+ @Test
+ public void testPartitionPushDown() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.smallData3()),
+ new String[] {"a int", "p bigint", "c varchar"},
+ "p",
+ new HashMap<String, String>() {
+ {
+ put("partition-list", "p:1;p:2");
+ }
+ });
+ File sinkPath = createSinkTable("MySink", "a int", "p bigint", "c varchar");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select * from MyTable where p = 2");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("2,2,Hello", "3,2,Hello world"), sinkPath);
+ }
+
+ @Test
+ public void testWatermarkPushDown() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.data3WithTimestamp()),
+ new String[] {
+ "a int",
+ "b bigint",
+ "c varchar",
+ "ts timestamp(3)",
+ "watermark for ts as ts - interval '5' second"
+ },
+ new HashMap<String, String>() {
+ {
+ put("enable-watermark-push-down", "true");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a int", "b bigint", "ts timestamp(3)");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select a, b, ts from MyTable where b = 3");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(
+ Arrays.asList(
+ "4,3," + formatSqlTimestamp(4000L),
+ "5,3," + formatSqlTimestamp(5000L),
+ "6,3," + formatSqlTimestamp(6000L)),
+ sinkPath);
+ }
+
+ @Test
+ public void testPushDowns() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.data3WithTimestamp()),
+ new String[] {
+ "a int",
+ "b bigint",
+ "c varchar",
+ "ts timestamp(3)",
+ "watermark for ts as ts - interval '5' second"
+ },
+ "b",
+ new HashMap<String, String>() {
+ {
+ put("readable-metadata", "a:INT");
+ put("filterable-fields", "a");
+ put("enable-watermark-push-down", "true");
+ put("partition-list", "b:1;b:2;b:3;b:4;b:5;b:6");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a int", "ts timestamp(3)");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan(
+ "insert into MySink select a, ts from MyTable where b = 3 and a > 4");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(
+ Arrays.asList("5," + formatSqlTimestamp(5000L), "6," + formatSqlTimestamp(6000L)),
+ sinkPath);
+ }
+
+ private static String formatSqlTimestamp(long timestamp) {
+ return TimeFormats.SQL_TIMESTAMP_FORMAT.format(unixTimestampToLocalDateTime(timestamp));
+ }
Review comment:
The method I'm talking about should be in `DateTimeUtils` in table-common
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 29e6025b084eb1c850ab3901c757dcfcb442c67e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289)
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
* 949b964bcb65db05a9c903a28f59f933300a7935 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r771375837
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @ClassRule public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
+
+ @Rule
Review comment:
`ClassRule`
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
Review comment:
+ writer test (but I know that you are working on it, so it's just a reminder for me)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e5445170476fae8081037143c62c36318677ad6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364)
* 0c1c04317c7fb3f4470cdf596756172105d25cda Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r773805659
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
Notice that `CsvSerDeSchemaFactory` is just a rename of the previously existing `CsvFormatFactory`, we did not touch its functionality in the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770679363
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##########
@@ -19,103 +19,48 @@
package org.apache.flink.formats.csv;
import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.common.serialization.SerializationSchema;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.configuration.ReadableConfig;
-import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.impl.StreamFormatAdapter;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.formats.common.Converter;
import org.apache.flink.table.connector.ChangelogMode;
-import org.apache.flink.table.connector.format.DecodingFormat;
-import org.apache.flink.table.connector.format.EncodingFormat;
-import org.apache.flink.table.connector.sink.DynamicTableSink;
-import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.Projection;
+import org.apache.flink.table.connector.format.BulkDecodingFormat;
+import org.apache.flink.table.connector.format.ProjectableDecodingFormat;
+import org.apache.flink.table.connector.source.DynamicTableSource.Context;
import org.apache.flink.table.data.RowData;
-import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.BulkReaderFormatFactory;
import org.apache.flink.table.factories.DynamicTableFactory;
-import org.apache.flink.table.factories.FactoryUtil;
-import org.apache.flink.table.factories.SerializationFormatFactory;
import org.apache.flink.table.types.DataType;
import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
import org.apache.commons.lang3.StringEscapeUtils;
import java.util.Collections;
-import java.util.HashSet;
import java.util.Set;
import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
-import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
-/**
- * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
- * and {@link DeserializationSchema}.
- */
+/** CSV format factory for file system. */
+//TODO: Rename back to CsvFileSystemFormatFactory? @Arvid
Review comment:
I had this thought because `BulkReader` is a `flink-connector-file` interface, and `CsvBulkWriter` writes into `FSDataOutputStream`. Both the encoder and the decoder exposed by the factory are for files.
I like the `TableCsvFormatFactory` - will rename it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155)
* 29e6025b084eb1c850ab3901c757dcfcb442c67e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770835661
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormat.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
Review comment:
Do you have a good inspirational example for this kind of instructions in our codebase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-996095786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770722990
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.unixTimestampToLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class CsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
+ public void testProjectPushDown() throws Exception {
+ List<String> data = Arrays.asList("1,1,hi", "2,1,hello", "3,2,hello world");
+ createSourceTable("MyTable", data, "a bigint", "b int not null", "c varchar");
+ File sinkPath = createSinkTable("MySink", "a bigint", "c varchar");
+
+ String jsonPlan = tableEnv.getJsonPlan("insert into MySink select a, c from MyTable");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("1,hi", "2,hello", "3,hello world"), sinkPath);
+ }
+
+ @Test
+ public void testReadingMetadata() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.smallData3()),
+ new String[] {"a int", "b bigint", "m varchar metadata"},
+ new HashMap<String, String>() {
+ {
+ put("readable-metadata", "m:STRING");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a bigint", "m varchar");
+
+ String jsonPlan = tableEnv.getJsonPlan("insert into MySink select a, m from MyTable");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("1,Hi", "2,Hello", "3,Hello world"), sinkPath);
+ }
+
+ @Test
+ public void testFilterPushDown() throws Exception {
+ List<String> data = Arrays.asList("1,1,hi", "2,1,hello", "3,2,hello world");
+ createSourceTable("MyTable", data, "a bigint", "b int not null", "c varchar");
+ File sinkPath = createSinkTable("MySink", "a bigint", "b int", "c varchar");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select * from MyTable where a > 1");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("2,1,hello", "3,2,hello world"), sinkPath);
+ }
+
+ @Test
+ public void testPartitionPushDown() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.smallData3()),
+ new String[] {"a int", "p bigint", "c varchar"},
+ "p",
+ new HashMap<String, String>() {
+ {
+ put("partition-list", "p:1;p:2");
+ }
+ });
+ File sinkPath = createSinkTable("MySink", "a int", "p bigint", "c varchar");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select * from MyTable where p = 2");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("2,2,Hello", "3,2,Hello world"), sinkPath);
+ }
+
+ @Test
+ public void testWatermarkPushDown() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.data3WithTimestamp()),
+ new String[] {
+ "a int",
+ "b bigint",
+ "c varchar",
+ "ts timestamp(3)",
+ "watermark for ts as ts - interval '5' second"
+ },
+ new HashMap<String, String>() {
+ {
+ put("enable-watermark-push-down", "true");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a int", "b bigint", "ts timestamp(3)");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select a, b, ts from MyTable where b = 3");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(
+ Arrays.asList(
+ "4,3," + formatSqlTimestamp(4000L),
+ "5,3," + formatSqlTimestamp(5000L),
+ "6,3," + formatSqlTimestamp(6000L)),
+ sinkPath);
+ }
+
+ @Test
+ public void testPushDowns() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.data3WithTimestamp()),
+ new String[] {
+ "a int",
+ "b bigint",
+ "c varchar",
+ "ts timestamp(3)",
+ "watermark for ts as ts - interval '5' second"
+ },
+ "b",
+ new HashMap<String, String>() {
+ {
+ put("readable-metadata", "a:INT");
+ put("filterable-fields", "a");
+ put("enable-watermark-push-down", "true");
+ put("partition-list", "b:1;b:2;b:3;b:4;b:5;b:6");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a int", "ts timestamp(3)");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan(
+ "insert into MySink select a, ts from MyTable where b = 3 and a > 4");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(
+ Arrays.asList("5," + formatSqlTimestamp(5000L), "6," + formatSqlTimestamp(6000L)),
+ sinkPath);
+ }
+
+ private static String formatSqlTimestamp(long timestamp) {
+ return TimeFormats.SQL_TIMESTAMP_FORMAT.format(unixTimestampToLocalDateTime(timestamp));
+ }
Review comment:
Found this addition after rebase, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9aa8115b9b3fa8cc853f7821c29f939870c79e5d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611)
* ee8fa611b135924ad3a083ff71dbcbc2e1895351 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17598: [WIP][FLINK-tbd][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9aa8115b9b3fa8cc853f7821c29f939870c79e5d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9aa8115b9b3fa8cc853f7821c29f939870c79e5d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611)
* ee8fa611b135924ad3a083ff71dbcbc2e1895351 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7e097463582d225c33b56c3d2f067a3954867a11 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599)
* 077423c8c94f59aade26c2c57001a4551a1b28af UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r778892987
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -82,41 +102,86 @@ public void testBoundedTextFileSourceWithJackson() throws Exception {
final File testDir = TMP_FOLDER.newFolder();
writeFile(testDir, "data.csv", CSV_LINES);
- final CsvReaderFormat<CitiesPojo> csvFormat = CsvReaderFormat.forPojo(CitiesPojo.class);
- final List<CitiesPojo> result = initializeSourceAndReadData(testDir, csvFormat);
-
- final CitiesPojo[] expected =
- new CitiesPojo[] {
- new CitiesPojo(
- "Berlin",
- new BigDecimal("52.5167"),
- new BigDecimal("13.3833"),
- "Germany",
- "DE",
- "Berlin",
- "primary",
- 3644826L),
- new CitiesPojo(
- "San Francisco",
- new BigDecimal("37.7562"),
- new BigDecimal("-122.4430"),
- "United States",
- "US",
- "California",
- "",
- 3592294L),
- new CitiesPojo(
- "Beijing",
- new BigDecimal("39.9050"),
- new BigDecimal("116.3914"),
- "China",
- "CN",
- "Beijing",
- "primary",
- 19433000L)
- };
-
- assertEquals(Arrays.asList(expected), result);
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(testDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+ final File outDir = TMP_FOLDER.newFolder();
+
+ StreamingFileSink<CityPojo> sink =
+ StreamingFileSink.forBulkFormat(
+ new Path(outDir.toURI()), new TestCsvBulkWriterFactory())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .withBucketCheckInterval(10L)
+ .withRollingPolicy(build())
+ .build();
+
+ OneInputStreamOperatorTestHarness<CityPojo, Object> testHarness =
+ new OneInputStreamOperatorTestHarness<>(new StreamSink<>(sink), 1, 1, 0);
+
+ testHarness.setup();
+ testHarness.open();
+
+ testHarness.processElement(new StreamRecord<>(pojos[0], 1L));
+ testHarness.processElement(new StreamRecord<>(pojos[1], 1L));
+ testHarness.processElement(new StreamRecord<>(pojos[2], 1L));
+ testHarness.snapshot(1L, 1L);
+ testHarness.notifyOfCompletedCheckpoint(2L);
+
+ Map<File, String> contents = getFileContentByPath(outDir);
+ Map.Entry<File, String> onlyEntry = contents.entrySet().iterator().next();
+ String[] result = onlyEntry.getValue().split("\n");
+
+ assertThat(result, is(CSV_LINES));
+ // assertThat(result, arrayContaining(CSV_LINES));
+ }
+
+ static Map<File, String> getFileContentByPath(File directory) throws IOException {
+ Map<File, String> contents = new HashMap<>(4);
+
+ final Collection<File> filesInBucket = FileUtils.listFiles(directory, null, true);
+ for (File file : filesInBucket) {
+ contents.put(file, FileUtils.readFileToString(file));
+ }
+ return contents;
+ }
+
+ static final class TestCsvBulkWriterFactory implements BulkWriter.Factory<CityPojo> {
Review comment:
Added native POJOs support in CsvBulkWriter.
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -82,41 +102,86 @@ public void testBoundedTextFileSourceWithJackson() throws Exception {
final File testDir = TMP_FOLDER.newFolder();
writeFile(testDir, "data.csv", CSV_LINES);
- final CsvReaderFormat<CitiesPojo> csvFormat = CsvReaderFormat.forPojo(CitiesPojo.class);
- final List<CitiesPojo> result = initializeSourceAndReadData(testDir, csvFormat);
-
- final CitiesPojo[] expected =
- new CitiesPojo[] {
- new CitiesPojo(
- "Berlin",
- new BigDecimal("52.5167"),
- new BigDecimal("13.3833"),
- "Germany",
- "DE",
- "Berlin",
- "primary",
- 3644826L),
- new CitiesPojo(
- "San Francisco",
- new BigDecimal("37.7562"),
- new BigDecimal("-122.4430"),
- "United States",
- "US",
- "California",
- "",
- 3592294L),
- new CitiesPojo(
- "Beijing",
- new BigDecimal("39.9050"),
- new BigDecimal("116.3914"),
- "China",
- "CN",
- "Beijing",
- "primary",
- 19433000L)
- };
-
- assertEquals(Arrays.asList(expected), result);
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(testDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+ final File outDir = TMP_FOLDER.newFolder();
+
+ StreamingFileSink<CityPojo> sink =
+ StreamingFileSink.forBulkFormat(
+ new Path(outDir.toURI()), new TestCsvBulkWriterFactory())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .withBucketCheckInterval(10L)
+ .withRollingPolicy(build())
+ .build();
+
+ OneInputStreamOperatorTestHarness<CityPojo, Object> testHarness =
+ new OneInputStreamOperatorTestHarness<>(new StreamSink<>(sink), 1, 1, 0);
+
+ testHarness.setup();
+ testHarness.open();
+
+ testHarness.processElement(new StreamRecord<>(pojos[0], 1L));
+ testHarness.processElement(new StreamRecord<>(pojos[1], 1L));
+ testHarness.processElement(new StreamRecord<>(pojos[2], 1L));
+ testHarness.snapshot(1L, 1L);
+ testHarness.notifyOfCompletedCheckpoint(2L);
+
+ Map<File, String> contents = getFileContentByPath(outDir);
+ Map.Entry<File, String> onlyEntry = contents.entrySet().iterator().next();
+ String[] result = onlyEntry.getValue().split("\n");
+
+ assertThat(result, is(CSV_LINES));
+ // assertThat(result, arrayContaining(CSV_LINES));
+ }
+
+ static Map<File, String> getFileContentByPath(File directory) throws IOException {
+ Map<File, String> contents = new HashMap<>(4);
+
+ final Collection<File> filesInBucket = FileUtils.listFiles(directory, null, true);
+ for (File file : filesInBucket) {
+ contents.put(file, FileUtils.readFileToString(file));
+ }
+ return contents;
+ }
+
+ static final class TestCsvBulkWriterFactory implements BulkWriter.Factory<CityPojo> {
+
+ private static final long serialVersionUID = 1L;
+
+ @Override
+ public BulkWriter<CityPojo> create(FSDataOutputStream out) {
+
+ final ObjectMapper objectMapper = new ObjectMapper();
+ final CsvMapper csvMapper = new CsvMapper();
+ final CsvSchema schema = csvMapper.schemaFor(CityPojo.class).withoutQuoteChar();
+
+ PojoConverter.ConverterContext converterContext =
+ new PojoConverter.ConverterContext(objectMapper);
+ final PojoConverter converter = createConverter();
+
+ return CsvBulkWriter.forSchema(csvMapper, schema, converter, converterContext, out);
+ }
+ }
+
+ interface PojoConverter extends Converter<CityPojo, JsonNode, PojoConverter.ConverterContext> {
+
+ class ConverterContext implements Serializable {
+ ObjectMapper mapper;
+
+ public ConverterContext(ObjectMapper csvMapper) {
+ this.mapper = csvMapper;
+ }
+ }
+ }
+
+ public static PojoConverter createConverter() {
+ return (pojo, context) -> context.mapper.valueToTree(pojo);
Review comment:
Added native POJOs support in CsvBulkWriter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 949b964bcb65db05a9c903a28f59f933300a7935 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337)
* 02355cdca67d73500f5251834bc07df18ec1f2a8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r779593652
##########
File path: flink-formats/flink-format-common/src/main/java/org/apache/flink/formats/common/Converter.java
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.common;
+
+import java.io.Serializable;
+
+/**
+ * A generic interface for converting data types.
+ *
+ * @param <From> The type of the element to be converted.
+ * @param <To> The output type.
+ * @param <C> The context for passing optional conversion instructions.
+ */
+public interface Converter<From, To, C> extends Serializable {
Review comment:
I think you have to mark it is as `@PublicEvolving` because it is used by your new CSV format that also is `@PublicEvolving`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
slinkydeveloper commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r780071456
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/TableCsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.toLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class TableCsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
Review comment:
Please let's keep `JsonPlanTestBase` and don't add a new test base. The test base you added is based on legacy as well (`TestValuesTableFactory`), and it's doing assertions based on strings while we want to use the new `TableAssertions`. So your new base will immediately become just another legacy to clean up later :smile:.
We're planning to do proper rework of our bases and push them in the new `flink-table-test-utils`, and these needs to be independent of table planner, so connector and formats stop depending on table planner test classpath. If you wanna create a new test base, feel free to start from this point: push it to `flink-table-test-utils` and use the new `TableAssertions`. But I'm pretty sure it's quite some work, so no need to do it in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc499251534da1f28801ef639c8eb8ee261de3c8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064)
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770126659
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormat.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
+ * @param <T>
+ */
+@PublicEvolving
+public class CsvFormat<T> extends SimpleStreamFormat<T> {
+
+ private static final long serialVersionUID = 1L;
+
+ private final CsvMapper mapper;
+ private final CsvSchema schema;
+ private final Class<Object> rootType;
+ private final Converter<Object, T, Void> converter;
+ private final TypeInformation<T> typeInformation;
+
+ @SuppressWarnings("unchecked")
+ <R> CsvFormat(
+ CsvMapper mapper,
+ CsvSchema schema,
+ Class<R> rootType,
+ Converter<R, T, Void> converter,
+ TypeInformation<T> typeInformation) {
+ this.mapper = checkNotNull(mapper);
+ this.schema = schema;
+ this.rootType = (Class<Object>) checkNotNull(rootType);
+ this.typeInformation = checkNotNull(typeInformation);
+ this.converter = (Converter<Object, T, Void>) checkNotNull(converter);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} using a {@code CsvSchema}.
+ *
+ * @param schema The Jackson CSV schema configured for parsing specific CSV files.
+ * @param typeInformation The Flink type descriptor of the returned elements.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(CsvSchema schema, TypeInformation<T> typeInformation) {
+ return from(new CsvMapper(), schema, typeInformation);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} using a {@code CsvSchema} and a pre-created {@code CsvMapper}.
+ *
+ * @param mapper The pre-created {@code CsvMapper}.
+ * @param schema The Jackson CSV schema configured for parsing specific CSV files.
+ * @param typeInformation The Flink type descriptor of the returned elements.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(
+ CsvMapper mapper, CsvSchema schema, TypeInformation<T> typeInformation) {
+ return new CsvFormat<>(
+ mapper,
+ schema,
+ typeInformation.getTypeClass(),
+ (value, context) -> value,
+ typeInformation);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} for reading CSV files mapped to the provided POJO class
+ * definition.
+ *
+ * @param pojoType The type class of the POJO.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(Class<T> pojoType) {
+ CsvMapper mapper = new CsvMapper();
+ return from(mapper, mapper.schemaFor(pojoType), TypeInformation.of(pojoType));
+ }
+
+ @Override
+ public StreamFormat.Reader<T> createReader(Configuration config, FSDataInputStream stream)
+ throws IOException {
+ return new Reader<>(mapper.readerFor(rootType).with(schema).readValues(stream), converter);
+ }
+
+ @Override
+ public TypeInformation<T> getProducedType() {
+ return typeInformation;
+ }
+
+ // ------------------------------------------------------------------------
+
+ /** The actual reader for the {@code CsvFormat}. */
+ public static final class Reader<R, T> implements StreamFormat.Reader<T> {
Review comment:
Good point. Copy-paste error from the `TextLineFormat` format "template".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770161770
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
I guess it could.
@AHeise, I believe we did not previously discuss this aspect. Do you think it is necessary to expand the scope of projection pushdown support also to the Ser/De schema-based implementation for this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r771428588
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
Review comment:
Our current `CsvBulkWriter` is currently pinned to be used in the Table/SQL context because it relies on the `RowDataToCsvFormatConverterContext`. It seems that we might need a different approach given that using it in the DataStream API is also a requirement.
https://github.com/apache/flink/pull/17598/files#diff-a32dfa08e3b14134b196779c88529037e79fd77d350de57d7738dcd4d295f9eaR81
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 29e6025b084eb1c850ab3901c757dcfcb442c67e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289)
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
* 949b964bcb65db05a9c903a28f59f933300a7935 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r771554971
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -82,41 +102,86 @@ public void testBoundedTextFileSourceWithJackson() throws Exception {
final File testDir = TMP_FOLDER.newFolder();
writeFile(testDir, "data.csv", CSV_LINES);
- final CsvReaderFormat<CitiesPojo> csvFormat = CsvReaderFormat.forPojo(CitiesPojo.class);
- final List<CitiesPojo> result = initializeSourceAndReadData(testDir, csvFormat);
-
- final CitiesPojo[] expected =
- new CitiesPojo[] {
- new CitiesPojo(
- "Berlin",
- new BigDecimal("52.5167"),
- new BigDecimal("13.3833"),
- "Germany",
- "DE",
- "Berlin",
- "primary",
- 3644826L),
- new CitiesPojo(
- "San Francisco",
- new BigDecimal("37.7562"),
- new BigDecimal("-122.4430"),
- "United States",
- "US",
- "California",
- "",
- 3592294L),
- new CitiesPojo(
- "Beijing",
- new BigDecimal("39.9050"),
- new BigDecimal("116.3914"),
- "China",
- "CN",
- "Beijing",
- "primary",
- 19433000L)
- };
-
- assertEquals(Arrays.asList(expected), result);
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(testDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+ final File outDir = TMP_FOLDER.newFolder();
+
+ StreamingFileSink<CityPojo> sink =
+ StreamingFileSink.forBulkFormat(
+ new Path(outDir.toURI()), new TestCsvBulkWriterFactory())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .withBucketCheckInterval(10L)
+ .withRollingPolicy(build())
+ .build();
+
+ OneInputStreamOperatorTestHarness<CityPojo, Object> testHarness =
+ new OneInputStreamOperatorTestHarness<>(new StreamSink<>(sink), 1, 1, 0);
+
+ testHarness.setup();
+ testHarness.open();
+
+ testHarness.processElement(new StreamRecord<>(pojos[0], 1L));
+ testHarness.processElement(new StreamRecord<>(pojos[1], 1L));
+ testHarness.processElement(new StreamRecord<>(pojos[2], 1L));
+ testHarness.snapshot(1L, 1L);
+ testHarness.notifyOfCompletedCheckpoint(2L);
+
+ Map<File, String> contents = getFileContentByPath(outDir);
+ Map.Entry<File, String> onlyEntry = contents.entrySet().iterator().next();
+ String[] result = onlyEntry.getValue().split("\n");
+
+ assertThat(result, is(CSV_LINES));
+ // assertThat(result, arrayContaining(CSV_LINES));
+ }
+
+ static Map<File, String> getFileContentByPath(File directory) throws IOException {
+ Map<File, String> contents = new HashMap<>(4);
+
+ final Collection<File> filesInBucket = FileUtils.listFiles(directory, null, true);
+ for (File file : filesInBucket) {
+ contents.put(file, FileUtils.readFileToString(file));
+ }
+ return contents;
+ }
+
+ static final class TestCsvBulkWriterFactory implements BulkWriter.Factory<CityPojo> {
Review comment:
This feels a bit awkward, how about
```
class CsvBulkWriter {
static <T> BulkWriter.Factory<T> forPojo(Class<T> pojoClass) {
final ObjectMapper objectMapper = new ObjectMapper();
PojoConverter.ConverterContext converterContext =
new PojoConverter.ConverterContext(objectMapper);
final PojoConverter converter = createConverter(); // see next comment
final CsvMapper csvMapper = new CsvMapper();
final CsvSchema schema = csvMapper.schemaFor(pojoClass);
new CsvBulkWriter.Factory(schema, csvMapper, converter); // basically same ctor as CsvFormat with converter going into the opposite direction
}
}
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -82,41 +102,86 @@ public void testBoundedTextFileSourceWithJackson() throws Exception {
final File testDir = TMP_FOLDER.newFolder();
writeFile(testDir, "data.csv", CSV_LINES);
- final CsvReaderFormat<CitiesPojo> csvFormat = CsvReaderFormat.forPojo(CitiesPojo.class);
- final List<CitiesPojo> result = initializeSourceAndReadData(testDir, csvFormat);
-
- final CitiesPojo[] expected =
- new CitiesPojo[] {
- new CitiesPojo(
- "Berlin",
- new BigDecimal("52.5167"),
- new BigDecimal("13.3833"),
- "Germany",
- "DE",
- "Berlin",
- "primary",
- 3644826L),
- new CitiesPojo(
- "San Francisco",
- new BigDecimal("37.7562"),
- new BigDecimal("-122.4430"),
- "United States",
- "US",
- "California",
- "",
- 3592294L),
- new CitiesPojo(
- "Beijing",
- new BigDecimal("39.9050"),
- new BigDecimal("116.3914"),
- "China",
- "CN",
- "Beijing",
- "primary",
- 19433000L)
- };
-
- assertEquals(Arrays.asList(expected), result);
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(testDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+ final File outDir = TMP_FOLDER.newFolder();
+
+ StreamingFileSink<CityPojo> sink =
+ StreamingFileSink.forBulkFormat(
+ new Path(outDir.toURI()), new TestCsvBulkWriterFactory())
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .withBucketCheckInterval(10L)
+ .withRollingPolicy(build())
+ .build();
+
+ OneInputStreamOperatorTestHarness<CityPojo, Object> testHarness =
+ new OneInputStreamOperatorTestHarness<>(new StreamSink<>(sink), 1, 1, 0);
+
+ testHarness.setup();
+ testHarness.open();
+
+ testHarness.processElement(new StreamRecord<>(pojos[0], 1L));
+ testHarness.processElement(new StreamRecord<>(pojos[1], 1L));
+ testHarness.processElement(new StreamRecord<>(pojos[2], 1L));
+ testHarness.snapshot(1L, 1L);
+ testHarness.notifyOfCompletedCheckpoint(2L);
+
+ Map<File, String> contents = getFileContentByPath(outDir);
+ Map.Entry<File, String> onlyEntry = contents.entrySet().iterator().next();
+ String[] result = onlyEntry.getValue().split("\n");
+
+ assertThat(result, is(CSV_LINES));
+ // assertThat(result, arrayContaining(CSV_LINES));
+ }
+
+ static Map<File, String> getFileContentByPath(File directory) throws IOException {
+ Map<File, String> contents = new HashMap<>(4);
+
+ final Collection<File> filesInBucket = FileUtils.listFiles(directory, null, true);
+ for (File file : filesInBucket) {
+ contents.put(file, FileUtils.readFileToString(file));
+ }
+ return contents;
+ }
+
+ static final class TestCsvBulkWriterFactory implements BulkWriter.Factory<CityPojo> {
+
+ private static final long serialVersionUID = 1L;
+
+ @Override
+ public BulkWriter<CityPojo> create(FSDataOutputStream out) {
+
+ final ObjectMapper objectMapper = new ObjectMapper();
+ final CsvMapper csvMapper = new CsvMapper();
+ final CsvSchema schema = csvMapper.schemaFor(CityPojo.class).withoutQuoteChar();
+
+ PojoConverter.ConverterContext converterContext =
+ new PojoConverter.ConverterContext(objectMapper);
+ final PojoConverter converter = createConverter();
+
+ return CsvBulkWriter.forSchema(csvMapper, schema, converter, converterContext, out);
+ }
+ }
+
+ interface PojoConverter extends Converter<CityPojo, JsonNode, PojoConverter.ConverterContext> {
+
+ class ConverterContext implements Serializable {
+ ObjectMapper mapper;
+
+ public ConverterContext(ObjectMapper csvMapper) {
+ this.mapper = csvMapper;
+ }
+ }
+ }
+
+ public static PojoConverter createConverter() {
+ return (pojo, context) -> context.mapper.valueToTree(pojo);
Review comment:
Why do we need a special PojoConverter? On reading side, we just do `(value, context) -> value` for POJO, right? Shouldn't the converter already receive the mapped value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r771428588
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
Review comment:
Our current `CsvBulkWriter` is pinned to be used in the Table/SQL context because it relies on the `RowDataToCsvFormatConverterContext`. It seems that we might need a different approach if using it in the DataStream API is also a requirement.
https://github.com/apache/flink/pull/17598/files#diff-a32dfa08e3b14134b196779c88529037e79fd77d350de57d7738dcd4d295f9eaR81
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 744b699a82b4640652050eb75eecaf78b2f10c2c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 949b964bcb65db05a9c903a28f59f933300a7935 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r792685482
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFileFormatFactory.java
##########
@@ -109,17 +110,21 @@ public CsvBulkDecodingFormat(ReadableConfig formatOptions) {
final RowType physicalRowType = (RowType) physicalDataType.getLogicalType();
final CsvSchema schema = buildCsvSchema(physicalRowType, formatOptions);
+ final boolean ignoreParseErrors =
+ formatOptions.getOptional(IGNORE_PARSE_ERRORS).isPresent();
final Converter<JsonNode, RowData, Void> converter =
(Converter)
- new CsvToRowDataConverters(false)
+ new CsvToRowDataConverters(ignoreParseErrors)
.createRowConverter(projectedRowType, true);
- return new StreamFormatAdapter<>(
+ CsvReaderFormat<RowData> csvReaderFormat =
new CsvReaderFormat<>(
new CsvMapper(),
schema,
JsonNode.class,
converter,
- context.createTypeInformation(projectedDataType)));
+ context.createTypeInformation(projectedDataType));
+ csvReaderFormat.setIgnoreParseErrors(ignoreParseErrors);
Review comment:
It is a bit unclear what do you propose? Do you mean adding something like a `ReaderConfig` with a builder and passing it into the `CsvReaderFormat` instead of the individual parameters?
As mentioned in my previous comment, I decided to make it a setter because otherwise, we would need to expand all of the static factory methods (`fromShcma`, `fromPojo`) and "pollute" them with this special concern. In that case I would add utility "telescope" methods with this parameter set to "ignore-parse-errors=false" by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6aea9ddf28bf11ed35c87ef57e477fecca94f266 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192)
* 4e5445170476fae8081037143c62c36318677ad6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c1c04317c7fb3f4470cdf596756172105d25cda Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391)
* 7e097463582d225c33b56c3d2f067a3954867a11 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r789562890
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @TempDir File outDir;
+
+ public static final MiniClusterExtension MINI_CLUSTER_RESOURCE =
+ new MiniClusterExtension(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ @RegisterExtension
+ public static AllCallbackWrapper<MiniClusterExtension> allCallbackWrapper =
+ new AllCallbackWrapper<>(MINI_CLUSTER_RESOURCE);
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
+ public void testBoundedTextFileSourceWithJackson() throws Exception {
+ writeFile(outDir, "data.csv", CSV_LINES);
+
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(outDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
+ // fromCollection is not bounded, using fromSequence instead
+ final List<CityPojo> pojosList = Arrays.asList(pojos); // needs to be Serializable
+ final DataStream<Integer> sequence =
+ env.fromSequence(0, pojos.length - 1).map(Long::intValue);
+ final DataStream<CityPojo> stream = sequence.map(pojosList::get).returns(CityPojo.class);
+
+ FileSink<CityPojo> sink =
+ FileSink.forBulkFormat(new Path(outDir.toURI()), factoryForPojo(CityPojo.class))
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+
+ stream.sinkTo(sink);
+ env.execute();
+
+ String[] result = getResultsFromSinkFiles(outDir);
+
+ assertThat(result).containsExactlyInAnyOrder(CSV_LINES);
Review comment:
Not sure what you mean, the call is checking that the lines are there in any order (containsExactlyIn*Any*Order)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-986209712
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770140011
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvToRowDataConverters.java
##########
@@ -305,7 +311,7 @@ private CsvToRowDataConverter createArrayConverter(ArrayType arrayType) {
}
private static void validateArity(int expected, int actual, boolean ignoreParseErrors) {
- if (expected != actual && !ignoreParseErrors) {
+ if (expected > actual && !ignoreParseErrors) {
Review comment:
The `CsvFormat.Reader` read() method needs to read elements according to the full schema from the file but emmit elements with a potentially reduced number of rows according to the projection pushdown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
slinkydeveloper commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770609158
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
@AHeise But then why `CsvFormatFactory` returns a `ProjectableDecodingFormat`? It doesn't make sense, becuse either none of them are projectable or both are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770164551
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.unixTimestampToLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class CsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
+ public void testProjectPushDown() throws Exception {
+ List<String> data = Arrays.asList("1,1,hi", "2,1,hello", "3,2,hello world");
+ createSourceTable("MyTable", data, "a bigint", "b int not null", "c varchar");
+ File sinkPath = createSinkTable("MySink", "a bigint", "c varchar");
+
+ String jsonPlan = tableEnv.getJsonPlan("insert into MySink select a, c from MyTable");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("1,hi", "2,hello", "3,hello world"), sinkPath);
+ }
+
+ @Test
+ public void testReadingMetadata() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.smallData3()),
+ new String[] {"a int", "b bigint", "m varchar metadata"},
+ new HashMap<String, String>() {
+ {
+ put("readable-metadata", "m:STRING");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a bigint", "m varchar");
+
+ String jsonPlan = tableEnv.getJsonPlan("insert into MySink select a, m from MyTable");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("1,Hi", "2,Hello", "3,Hello world"), sinkPath);
+ }
+
+ @Test
+ public void testFilterPushDown() throws Exception {
+ List<String> data = Arrays.asList("1,1,hi", "2,1,hello", "3,2,hello world");
+ createSourceTable("MyTable", data, "a bigint", "b int not null", "c varchar");
+ File sinkPath = createSinkTable("MySink", "a bigint", "b int", "c varchar");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select * from MyTable where a > 1");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("2,1,hello", "3,2,hello world"), sinkPath);
+ }
+
+ @Test
+ public void testPartitionPushDown() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.smallData3()),
+ new String[] {"a int", "p bigint", "c varchar"},
+ "p",
+ new HashMap<String, String>() {
+ {
+ put("partition-list", "p:1;p:2");
+ }
+ });
+ File sinkPath = createSinkTable("MySink", "a int", "p bigint", "c varchar");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select * from MyTable where p = 2");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("2,2,Hello", "3,2,Hello world"), sinkPath);
+ }
+
+ @Test
+ public void testWatermarkPushDown() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.data3WithTimestamp()),
+ new String[] {
+ "a int",
+ "b bigint",
+ "c varchar",
+ "ts timestamp(3)",
+ "watermark for ts as ts - interval '5' second"
+ },
+ new HashMap<String, String>() {
+ {
+ put("enable-watermark-push-down", "true");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a int", "b bigint", "ts timestamp(3)");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select a, b, ts from MyTable where b = 3");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(
+ Arrays.asList(
+ "4,3," + formatSqlTimestamp(4000L),
+ "5,3," + formatSqlTimestamp(5000L),
+ "6,3," + formatSqlTimestamp(6000L)),
+ sinkPath);
+ }
+
+ @Test
+ public void testPushDowns() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.data3WithTimestamp()),
+ new String[] {
+ "a int",
+ "b bigint",
+ "c varchar",
+ "ts timestamp(3)",
+ "watermark for ts as ts - interval '5' second"
+ },
+ "b",
+ new HashMap<String, String>() {
+ {
+ put("readable-metadata", "a:INT");
+ put("filterable-fields", "a");
+ put("enable-watermark-push-down", "true");
+ put("partition-list", "b:1;b:2;b:3;b:4;b:5;b:6");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a int", "ts timestamp(3)");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan(
+ "insert into MySink select a, ts from MyTable where b = 3 and a > 4");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(
+ Arrays.asList("5," + formatSqlTimestamp(5000L), "6," + formatSqlTimestamp(6000L)),
+ sinkPath);
+ }
+
+ private static String formatSqlTimestamp(long timestamp) {
+ return TimeFormats.SQL_TIMESTAMP_FORMAT.format(unixTimestampToLocalDateTime(timestamp));
+ }
Review comment:
Not sure what you mean. I can see the `formatUnixTimestamp` only in JavaScript libraries in Flink's code base. Or is your point that the timezone should necessarily be also taken into account and `formatUnixTimestamp(long unixtime, TimeZone tz)` is what you propose to call my method instead of formatSqlTimestamp?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770164551
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.unixTimestampToLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class CsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
+ public void testProjectPushDown() throws Exception {
+ List<String> data = Arrays.asList("1,1,hi", "2,1,hello", "3,2,hello world");
+ createSourceTable("MyTable", data, "a bigint", "b int not null", "c varchar");
+ File sinkPath = createSinkTable("MySink", "a bigint", "c varchar");
+
+ String jsonPlan = tableEnv.getJsonPlan("insert into MySink select a, c from MyTable");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("1,hi", "2,hello", "3,hello world"), sinkPath);
+ }
+
+ @Test
+ public void testReadingMetadata() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.smallData3()),
+ new String[] {"a int", "b bigint", "m varchar metadata"},
+ new HashMap<String, String>() {
+ {
+ put("readable-metadata", "m:STRING");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a bigint", "m varchar");
+
+ String jsonPlan = tableEnv.getJsonPlan("insert into MySink select a, m from MyTable");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("1,Hi", "2,Hello", "3,Hello world"), sinkPath);
+ }
+
+ @Test
+ public void testFilterPushDown() throws Exception {
+ List<String> data = Arrays.asList("1,1,hi", "2,1,hello", "3,2,hello world");
+ createSourceTable("MyTable", data, "a bigint", "b int not null", "c varchar");
+ File sinkPath = createSinkTable("MySink", "a bigint", "b int", "c varchar");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select * from MyTable where a > 1");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("2,1,hello", "3,2,hello world"), sinkPath);
+ }
+
+ @Test
+ public void testPartitionPushDown() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.smallData3()),
+ new String[] {"a int", "p bigint", "c varchar"},
+ "p",
+ new HashMap<String, String>() {
+ {
+ put("partition-list", "p:1;p:2");
+ }
+ });
+ File sinkPath = createSinkTable("MySink", "a int", "p bigint", "c varchar");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select * from MyTable where p = 2");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("2,2,Hello", "3,2,Hello world"), sinkPath);
+ }
+
+ @Test
+ public void testWatermarkPushDown() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.data3WithTimestamp()),
+ new String[] {
+ "a int",
+ "b bigint",
+ "c varchar",
+ "ts timestamp(3)",
+ "watermark for ts as ts - interval '5' second"
+ },
+ new HashMap<String, String>() {
+ {
+ put("enable-watermark-push-down", "true");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a int", "b bigint", "ts timestamp(3)");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select a, b, ts from MyTable where b = 3");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(
+ Arrays.asList(
+ "4,3," + formatSqlTimestamp(4000L),
+ "5,3," + formatSqlTimestamp(5000L),
+ "6,3," + formatSqlTimestamp(6000L)),
+ sinkPath);
+ }
+
+ @Test
+ public void testPushDowns() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.data3WithTimestamp()),
+ new String[] {
+ "a int",
+ "b bigint",
+ "c varchar",
+ "ts timestamp(3)",
+ "watermark for ts as ts - interval '5' second"
+ },
+ "b",
+ new HashMap<String, String>() {
+ {
+ put("readable-metadata", "a:INT");
+ put("filterable-fields", "a");
+ put("enable-watermark-push-down", "true");
+ put("partition-list", "b:1;b:2;b:3;b:4;b:5;b:6");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a int", "ts timestamp(3)");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan(
+ "insert into MySink select a, ts from MyTable where b = 3 and a > 4");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(
+ Arrays.asList("5," + formatSqlTimestamp(5000L), "6," + formatSqlTimestamp(6000L)),
+ sinkPath);
+ }
+
+ private static String formatSqlTimestamp(long timestamp) {
+ return TimeFormats.SQL_TIMESTAMP_FORMAT.format(unixTimestampToLocalDateTime(timestamp));
+ }
Review comment:
Not sure what you mean. I can see the `formatUnixTimestamp` only in JavaScript libraries in Flink's code base. Or is your point that the timezone should necessarily be also taken into account?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r773800347
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
@slinkydeveloper
I am not sure about this actually. The focus of this PR is
**Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.**
I would rather not touch the existing Ser/De schema-based functionality in this scope. Nothing will break if we won't support pushdown in the non-filesystem connectors for now and to me, it sounds like an improvement that should be done in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0e3bc85342e7ae33117597931baa83f7c78c19a4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687)
* 47b6d1194869dc00a6d326bf87127da8c6d667a3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47b6d1194869dc00a6d326bf87127da8c6d667a3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045)
* 6aea9ddf28bf11ed35c87ef57e477fecca94f266 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47b6d1194869dc00a6d326bf87127da8c6d667a3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045)
* 6aea9ddf28bf11ed35c87ef57e477fecca94f266 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r778924754
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectWriter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import java.io.IOException;
+
+// TODO: ? PublicEvolving?
+/** A simple {@link BulkWriter} implementation based on Jackson CSV transformations. */
+class CsvBulkWriter<T, R, C> implements BulkWriter<T> {
+
+ private final FSDataOutputStream stream;
+ private final Converter<T, R, C> converter;
+ private final C converterContext;
+ private final ObjectWriter csvWriter;
+
+ CsvBulkWriter(
+ CsvMapper mapper,
+ CsvSchema schema,
+ Converter<T, R, C> converter,
+ C converterContext,
+ FSDataOutputStream stream) {
+ this.converter = converter;
+ this.converterContext = converterContext;
+ this.stream = stream;
+
+ this.csvWriter = mapper.writer(schema);
+
+ // Prevent Jackson's writeValue() method calls from closing the stream.
+ mapper.getFactory().disable(JsonGenerator.Feature.AUTO_CLOSE_TARGET);
Review comment:
Nit: Can you try to use `Preconditions.checkNotNull()` for not nullable ctor arguments?
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
+ * @param <T> The type of the returned elements.
+ */
+@PublicEvolving
+public class CsvReaderFormat<T> extends SimpleStreamFormat<T> {
+
+ private static final long serialVersionUID = 1L;
+
+ private final CsvMapper mapper;
+ private final CsvSchema schema;
+ private final Class<Object> rootType;
+ private final Converter<Object, T, Void> converter;
+ private final TypeInformation<T> typeInformation;
+
+ @SuppressWarnings("unchecked")
+ <R> CsvReaderFormat(
+ CsvMapper mapper,
+ CsvSchema schema,
+ Class<R> rootType,
+ Converter<R, T, Void> converter,
+ TypeInformation<T> typeInformation) {
+ this.mapper = checkNotNull(mapper);
+ this.schema = schema;
Review comment:
Nit: checkNotNull
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.StreamSink;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy.build;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @ClassRule public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
+
+ @ClassRule
+ public static final MiniClusterWithClientResource miniClusterResource =
+ new MiniClusterWithClientResource(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
Review comment:
I am a bit confused by this class. Is this mini-cluster even used? After looking more thoroughly this class seems more like a normal unit test rather than an IT case
##########
File path: flink-formats/flink-format-common/src/main/java/org/apache/flink/formats/common/Converter.java
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.common;
+
+import java.io.Serializable;
+
+/**
+ * A generic interface for converting data types.
+ *
+ * @param <From> The type of the element to be converted.
+ * @param <To> The output type.
+ * @param <C> The context for passing optional conversion instructions.
+ */
+public interface Converter<From, To, C> extends Serializable {
Review comment:
I guess this interface is supposed to be `@PublicEvolving` or `@Experimental` since it might be used in the future by a lot of formats.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvCommons.java
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.ValidationException;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/** A class with common CSV format constants and utility methods. */
+@Internal
Review comment:
You don't need annotations for package-private classes.
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.StreamSink;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy.build;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @ClassRule public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
+
+ @ClassRule
+ public static final MiniClusterWithClientResource miniClusterResource =
+ new MiniClusterWithClientResource(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
+ public void testBoundedTextFileSourceWithJackson() throws Exception {
+ final File testDir = TMP_FOLDER.newFolder();
+ writeFile(testDir, "data.csv", CSV_LINES);
+
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(testDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+ final File outDir = TMP_FOLDER.newFolder();
+
+ StreamingFileSink<CityPojo> sink =
+ StreamingFileSink.forBulkFormat(
+ new Path(outDir.toURI()), factoryForPojo(CityPojo.class))
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .withBucketCheckInterval(10L)
+ .withRollingPolicy(build())
+ .build();
+
+ OneInputStreamOperatorTestHarness<CityPojo, Object> testHarness =
+ new OneInputStreamOperatorTestHarness<>(new StreamSink<>(sink), 1, 1, 0);
+
+ testHarness.setup();
+ testHarness.open();
+
+ testHarness.processElement(new StreamRecord<>(pojos[0], 1L));
+ testHarness.processElement(new StreamRecord<>(pojos[1], 1L));
+ testHarness.processElement(new StreamRecord<>(pojos[2], 1L));
+ testHarness.snapshot(1L, 1L);
+ testHarness.notifyOfCompletedCheckpoint(2L);
+
+ Map<File, String> contents = getFileContentByPath(outDir);
+ Map.Entry<File, String> onlyEntry = contents.entrySet().iterator().next();
+ String[] result = onlyEntry.getValue().split("\n");
+
+ assertThat(result, is(CSV_LINES));
+ }
+
+ static <T> BulkWriter.Factory<T> factoryForPojo(Class<T> pojoClass) {
Review comment:
private?
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/TableCsvFormatFactory.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.api.common.serialization.BulkWriter.Factory;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.impl.StreamFormatAdapter;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.connector.file.table.factories.BulkReaderFormatFactory;
+import org.apache.flink.connector.file.table.factories.BulkWriterFormatFactory;
+import org.apache.flink.connector.file.table.format.BulkDecodingFormat;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.formats.csv.RowDataToCsvConverters.RowDataToCsvConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.Projection;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.format.ProjectableDecodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource.Context;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+//TODO: clarify scope for naming Table/File:
+//https://github.com/apache/flink/pull/17598#discussion_r768753623
+/** CSV format factory for file system. */
+@Internal
+public class TableCsvFormatFactory implements BulkReaderFormatFactory, BulkWriterFormatFactory {
+
+ @Override
+ public String factoryIdentifier() {
+ return CsvCommons.IDENTIFIER;
+ }
+
+ @Override
+ public Set<ConfigOption<?>> requiredOptions() {
+ return Collections.emptySet();
+ }
+
+ @Override
+ public Set<ConfigOption<?>> optionalOptions() {
+ return CsvCommons.optionalOptions();
+ }
+
+ @Override
+ public BulkDecodingFormat<RowData> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+
+ return new CsvBulkDecodingFormat(formatOptions);
+ }
+
+ private static class CsvBulkDecodingFormat
+ implements BulkDecodingFormat<RowData>,
+ ProjectableDecodingFormat<BulkFormat<RowData, FileSourceSplit>> {
+
+ private final ReadableConfig formatOptions;
+
+ public CsvBulkDecodingFormat(ReadableConfig formatOptions) {
+ this.formatOptions = formatOptions;
Review comment:
nit: checkNotNull
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.StreamSink;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy.build;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @ClassRule public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
+
+ @ClassRule
+ public static final MiniClusterWithClientResource miniClusterResource =
+ new MiniClusterWithClientResource(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
Review comment:
Do you also want to add an ITCase for `CsvReaderFormat.fromSchema`?
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/TableCsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.toLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class TableCsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
Review comment:
Same as for the other test class can you use junit 5 and assertJ, please.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectWriter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import java.io.IOException;
+
+// TODO: ? PublicEvolving?
+/** A simple {@link BulkWriter} implementation based on Jackson CSV transformations. */
Review comment:
Is this TODO still up to date? The class is package-private so PublicEvolving feels unlikely
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.StreamSink;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy.build;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @ClassRule public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
+
+ @ClassRule
+ public static final MiniClusterWithClientResource miniClusterResource =
+ new MiniClusterWithClientResource(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
+ public void testBoundedTextFileSourceWithJackson() throws Exception {
+ final File testDir = TMP_FOLDER.newFolder();
+ writeFile(testDir, "data.csv", CSV_LINES);
+
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(testDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+ final File outDir = TMP_FOLDER.newFolder();
+
+ StreamingFileSink<CityPojo> sink =
+ StreamingFileSink.forBulkFormat(
+ new Path(outDir.toURI()), factoryForPojo(CityPojo.class))
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .withBucketCheckInterval(10L)
+ .withRollingPolicy(build())
+ .build();
+
+ OneInputStreamOperatorTestHarness<CityPojo, Object> testHarness =
+ new OneInputStreamOperatorTestHarness<>(new StreamSink<>(sink), 1, 1, 0);
+
+ testHarness.setup();
+ testHarness.open();
+
+ testHarness.processElement(new StreamRecord<>(pojos[0], 1L));
+ testHarness.processElement(new StreamRecord<>(pojos[1], 1L));
+ testHarness.processElement(new StreamRecord<>(pojos[2], 1L));
+ testHarness.snapshot(1L, 1L);
+ testHarness.notifyOfCompletedCheckpoint(2L);
+
+ Map<File, String> contents = getFileContentByPath(outDir);
+ Map.Entry<File, String> onlyEntry = contents.entrySet().iterator().next();
+ String[] result = onlyEntry.getValue().split("\n");
+
+ assertThat(result, is(CSV_LINES));
+ }
+
+ static <T> BulkWriter.Factory<T> factoryForPojo(Class<T> pojoClass) {
+ final Converter<T, T, Void> converter = (value, context) -> value;
+ final CsvMapper csvMapper = new CsvMapper();
+ final CsvSchema schema = csvMapper.schemaFor(pojoClass).withoutQuoteChar();
+ return (out) -> new CsvBulkWriter<>(csvMapper, schema, converter, null, out);
+ }
+
+ static Map<File, String> getFileContentByPath(File directory) throws IOException {
Review comment:
private?
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.StreamSink;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy.build;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @ClassRule public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
+
+ @ClassRule
Review comment:
Can you change the newly added tests to use junit 5 and assertJ?
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/TableCsvFormatFactory.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.api.common.serialization.BulkWriter.Factory;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.impl.StreamFormatAdapter;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.connector.file.table.factories.BulkReaderFormatFactory;
+import org.apache.flink.connector.file.table.factories.BulkWriterFormatFactory;
+import org.apache.flink.connector.file.table.format.BulkDecodingFormat;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.formats.csv.RowDataToCsvConverters.RowDataToCsvConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.Projection;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.format.ProjectableDecodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource.Context;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+//TODO: clarify scope for naming Table/File:
+//https://github.com/apache/flink/pull/17598#discussion_r768753623
Review comment:
TODO?
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectWriter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import java.io.IOException;
+
+// TODO: ? PublicEvolving?
+/** A simple {@link BulkWriter} implementation based on Jackson CSV transformations. */
+class CsvBulkWriter<T, R, C> implements BulkWriter<T> {
+
+ private final FSDataOutputStream stream;
+ private final Converter<T, R, C> converter;
+ private final C converterContext;
+ private final ObjectWriter csvWriter;
+
+ CsvBulkWriter(
+ CsvMapper mapper,
+ CsvSchema schema,
+ Converter<T, R, C> converter,
+ C converterContext,
+ FSDataOutputStream stream) {
+ this.converter = converter;
+ this.converterContext = converterContext;
+ this.stream = stream;
Review comment:
Where is the stream closed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc499251534da1f28801ef639c8eb8ee261de3c8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* e089768e55137235bd5b85a0b6b97b00dd0d23f9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r779596112
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.StreamSink;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy.build;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @ClassRule public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
+
+ @ClassRule
+ public static final MiniClusterWithClientResource miniClusterResource =
+ new MiniClusterWithClientResource(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
Review comment:
It is actually also tested implicitly via `fromPojo` https://github.com/afedulov/flink/blob/02355cdca67d73500f5251834bc07df18ec1f2a8/flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java#L109
Apart from the native schema construction via Jackson there seems not much be changed in the additional test. Or do you see it differently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r780379983
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/TableCsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.toLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class TableCsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
Review comment:
Reverted back to the `JsonPlanTestBase`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
slinkydeveloper commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r780071456
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/TableCsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.toLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class TableCsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
Review comment:
Please let's keep `JsonPlanTestBase` and don't add a new test base. The test base you added is based on legacy as well (`TestValuesTableFactory`), and it's doing assertions based on strings while we want to use the new `TableAssertions`. So your new base will immediately become just another legacy to clean up later :smile:.
We're planning to do proper rework of our bases and push them in the new `flink-table-test-utils`, and these needs to be independent of table planner, so connector and formats stop depending on table planner test classpath. If you wanna create a new test base, feel free to start from this point: push it to `flink-table-test-utils` and use the new `TableAssertions`. But I'm pretty sure it's quite some work, so no need to do it in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r784874977
##########
File path: flink-formats/flink-csv/pom.xml
##########
@@ -77,6 +77,14 @@ under the License.
<!-- test dependencies -->
+ <dependency>
Review comment:
Nit: Why was needed?
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectWriter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** A simple {@link BulkWriter} implementation based on Jackson CSV transformations. */
+class CsvBulkWriter<T, R, C> implements BulkWriter<T> {
+
+ private final FSDataOutputStream stream;
+ private final Converter<T, R, C> converter;
+ private final C converterContext;
+ private final ObjectWriter csvWriter;
+
+ CsvBulkWriter(
+ CsvMapper mapper,
+ CsvSchema schema,
+ Converter<T, R, C> converter,
+ @Nullable C converterContext,
+ FSDataOutputStream stream) {
+ checkNotNull(mapper);
+ checkNotNull(schema);
+ checkNotNull(converter);
+ checkNotNull(stream);
Review comment:
Nit: `checkNotNull` returns the value if not null so you could do:
```java
this.converter = checkNotNull(converter)
```
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @TempDir File outDir;
+
+ public static final MiniClusterExtension MINI_CLUSTER_RESOURCE =
Review comment:
Does this need to be public?
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @TempDir File outDir;
+
+ public static final MiniClusterExtension MINI_CLUSTER_RESOURCE =
+ new MiniClusterExtension(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ @RegisterExtension
+ public static AllCallbackWrapper<MiniClusterExtension> allCallbackWrapper =
Review comment:
Public required?
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvCommons.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.ValidationException;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/** A class with common CSV format constants and utility methods. */
+class CsvCommons {
+
+ public static final String IDENTIFIER = "csv";
+
+ // ------------------------------------------------------------------------
+ // Validation
+ // ------------------------------------------------------------------------
+
+ static void validateFormatOptions(ReadableConfig tableOptions) {
+ final boolean hasQuoteCharacter = tableOptions.getOptional(QUOTE_CHARACTER).isPresent();
+ final boolean isDisabledQuoteCharacter = tableOptions.get(DISABLE_QUOTE_CHARACTER);
+ if (isDisabledQuoteCharacter && hasQuoteCharacter) {
+ throw new ValidationException(
+ "Format cannot define a quote character and disabled quote character at the same time.");
+ }
+ // Validate the option value must be a single char.
+ validateCharacterVal(tableOptions, FIELD_DELIMITER, true);
+ validateCharacterVal(tableOptions, ARRAY_ELEMENT_DELIMITER);
+ validateCharacterVal(tableOptions, QUOTE_CHARACTER);
+ validateCharacterVal(tableOptions, ESCAPE_CHARACTER);
+ }
+
+ /** Validates the option {@code option} value must be a Character. */
+ private static void validateCharacterVal(
+ ReadableConfig tableOptions, ConfigOption<String> option) {
+ validateCharacterVal(tableOptions, option, false);
+ }
+
+ /**
+ * Validates the option {@code option} value must be a Character.
+ *
+ * @param tableOptions the table options
+ * @param option the config option
+ * @param unescape whether to unescape the option value
+ */
+ private static void validateCharacterVal(
+ ReadableConfig tableOptions, ConfigOption<String> option, boolean unescape) {
+ if (tableOptions.getOptional(option).isPresent()) {
Review comment:
Nit: I prefer explicit condition handling to improve readability.
```java
if (!tableOptions.getOptional(option).isPresent()) {
return;
}
...
```
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
+ * <p>The following example shows how to create a {@code CsvReaderFormat} where the schema for CSV
+ * parsing is automatically derived based on the fields of a POJO class.
+ *
+ * <pre>{@code
+ * CsvReaderFormat<SomePojo> csvFormat = CsvReaderFormat.forPojo(SomePojo.class);
+ * FileSource<SomePojo> source =
+ * FileSource.forRecordStreamFormat(csvFormat, Path.fromLocalFile(filesPath)).build();
+ * }</pre>
+ *
+ * <i> Note: you might need to add {@code @JsonPropertyOrder({field1, field2, ...})} annotation from
+ * the {@code Jackson} library to your class definition with the fields order exactly matching those
+ * of the CSV file columns).</i>
+ *
+ * <p>If you need more fine-grained control over the CSV schema or the parsing options, use the more
+ * low-level {@code forSchema} static factory method based on the {@code Jackson} library utilities:
+ *
+ * <pre>{@code
+ * CsvReaderFormat<T> forSchema(CsvMapper mapper,
Review comment:
Nit: The example does not look very helpful. Maybe you can show a code snipped in conjunction with using the format with the FileSource and it would also be great two a second example in the same code block showing it for a POJO class.
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @TempDir File outDir;
+
+ public static final MiniClusterExtension MINI_CLUSTER_RESOURCE =
+ new MiniClusterExtension(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ @RegisterExtension
+ public static AllCallbackWrapper<MiniClusterExtension> allCallbackWrapper =
+ new AllCallbackWrapper<>(MINI_CLUSTER_RESOURCE);
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
+ public void testBoundedTextFileSourceWithJackson() throws Exception {
+ writeFile(outDir, "data.csv", CSV_LINES);
+
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(outDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
+ // fromCollection is not bounded, using fromSequence instead
+ final List<CityPojo> pojosList = Arrays.asList(pojos); // needs to be Serializable
+ final DataStream<Integer> sequence =
+ env.fromSequence(0, pojos.length - 1).map(Long::intValue);
+ final DataStream<CityPojo> stream = sequence.map(pojosList::get).returns(CityPojo.class);
+
+ FileSink<CityPojo> sink =
+ FileSink.forBulkFormat(new Path(outDir.toURI()), factoryForPojo(CityPojo.class))
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+
+ stream.sinkTo(sink);
+ env.execute();
+
+ String[] result = getResultsFromSinkFiles(outDir);
+
+ assertThat(result).containsExactlyInAnyOrder(CSV_LINES);
Review comment:
Is the order really guaranteed if we run with higher parallelism than 1? I would expect parallel threads writing to the file so they cannot ensure the ordering.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectWriter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** A simple {@link BulkWriter} implementation based on Jackson CSV transformations. */
+class CsvBulkWriter<T, R, C> implements BulkWriter<T> {
+
+ private final FSDataOutputStream stream;
+ private final Converter<T, R, C> converter;
+ private final C converterContext;
Review comment:
Nit: annotate with nullable
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @TempDir File outDir;
+
+ public static final MiniClusterExtension MINI_CLUSTER_RESOURCE =
+ new MiniClusterExtension(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ @RegisterExtension
+ public static AllCallbackWrapper<MiniClusterExtension> allCallbackWrapper =
+ new AllCallbackWrapper<>(MINI_CLUSTER_RESOURCE);
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
+ public void testBoundedTextFileSourceWithJackson() throws Exception {
+ writeFile(outDir, "data.csv", CSV_LINES);
+
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(outDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
+ // fromCollection is not bounded, using fromSequence instead
+ final List<CityPojo> pojosList = Arrays.asList(pojos); // needs to be Serializable
+ final DataStream<Integer> sequence =
+ env.fromSequence(0, pojos.length - 1).map(Long::intValue);
+ final DataStream<CityPojo> stream = sequence.map(pojosList::get).returns(CityPojo.class);
+
+ FileSink<CityPojo> sink =
+ FileSink.forBulkFormat(new Path(outDir.toURI()), factoryForPojo(CityPojo.class))
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+
+ stream.sinkTo(sink);
+ env.execute();
+
+ String[] result = getResultsFromSinkFiles(outDir);
+
+ assertThat(result).containsExactlyInAnyOrder(CSV_LINES);
+ }
+
+ @NotNull
+ private String[] getResultsFromSinkFiles(File outDir) throws IOException {
+ final Map<File, String> contents = getFileContentByPath(outDir);
+
+ List<String> resultList =
+ contents.entrySet().stream()
+ .flatMap(e -> Arrays.stream(e.getValue().split("\n")))
+ .collect(Collectors.toList());
+
+ String[] result = resultList.toArray(new String[0]);
+ return result;
+ }
+
+ private static <T> BulkWriter.Factory<T> factoryForPojo(Class<T> pojoClass) {
+ final Converter<T, T, Void> converter = (value, context) -> value;
+ final CsvMapper csvMapper = new CsvMapper();
+ final CsvSchema schema = csvMapper.schemaFor(pojoClass).withoutQuoteChar();
+ return (out) -> new CsvBulkWriter<>(csvMapper, schema, converter, null, out);
+ }
+
+ private static Map<File, String> getFileContentByPath(File directory) throws IOException {
+ Map<File, String> contents = new HashMap<>(4);
+
+ final Collection<File> filesInBucket = FileUtils.listFiles(directory, null, true);
+ for (File file : filesInBucket) {
+ contents.put(file, FileUtils.readFileToString(file));
+ }
+ return contents;
+ }
+
+ @JsonPropertyOrder({
+ "city",
+ "lat",
+ "lng",
+ "country",
+ "iso2",
+ "adminName",
+ "capital",
+ "population"
+ })
+ public static class CityPojo implements Serializable {
+ public String city;
+ public BigDecimal lat;
+ public BigDecimal lng;
+ public String country;
+ public String iso2;
+ public String adminName;
+ public String capital;
+ public long population;
+
+ public CityPojo() {};
+
+ public CityPojo(
+ String city,
+ BigDecimal lat,
+ BigDecimal lng,
+ String country,
+ String iso2,
+ String admin_name,
+ String capital,
+ long population) {
+ this.city = city;
+ this.lat = lat;
+ this.lng = lng;
+ this.country = country;
+ this.iso2 = iso2;
+ this.adminName = admin_name;
+ this.capital = capital;
+ this.population = population;
+ }
+
+ @Override
+ public String toString() {
+ return "CitiesPojo{"
+ + "city='"
+ + city
+ + '\''
+ + ", lat="
+ + lat
+ + ", lng="
+ + lng
+ + ", country='"
+ + country
+ + '\''
+ + ", iso2='"
+ + iso2
+ + '\''
+ + ", adminName='"
+ + adminName
+ + '\''
+ + ", capital='"
+ + capital
+ + '\''
+ + ", population="
+ + population
+ + '}';
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ CityPojo that = (CityPojo) o;
+ return population == that.population
+ && Objects.equals(city, that.city)
+ && Objects.equals(lat, that.lat)
+ && Objects.equals(lng, that.lng)
+ && Objects.equals(country, that.country)
+ && Objects.equals(iso2, that.iso2)
+ && Objects.equals(adminName, that.adminName)
+ && Objects.equals(capital, that.capital);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(city, lat, lng, country, iso2, adminName, capital, population);
+ }
+ }
+
+ private static <T> List<T> initializeSourceAndReadData(File testDir, StreamFormat<T> csvFormat)
+ throws Exception {
+ final FileSource<T> source =
+ FileSource.forRecordStreamFormat(csvFormat, Path.fromLocalFile(testDir)).build();
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
+ final DataStream<T> stream =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "file-source");
+
+ return getResultsFromStream(stream);
+ }
+
+ @NotNull
+ private static <T> List<T> getResultsFromStream(DataStream<T> stream) throws Exception {
+ final ClientAndIterator<T> client =
+ DataStreamUtils.collectWithClient(stream, "Bounded Results Fetch");
+
+ final List<T> result = new ArrayList<>();
+ while (client.iterator.hasNext()) {
+ T next = client.iterator.next();
+ result.add(next);
+ }
+ return result;
+ }
+
+ // ------------------------------------------------------------------------
+ // test data
+ // ------------------------------------------------------------------------
+
+ private static final String[] CSV_LINES =
+ new String[] {
+ "Berlin,52.5167,13.3833,Germany,DE,Berlin,primary,3644826",
+ "San Francisco,37.7562,-122.443,United States,US,California,,3592294",
+ "Beijing,39.905,116.3914,China,CN,Beijing,primary,19433000"
+ };
+
+ final CityPojo[] pojos =
Review comment:
Nit: static
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
Review comment:
You are using the `TestLogger` for junit4. Please replace it with the `TestLoggerExtension` [1]
[1] https://github.com/apache/flink/blob/3fe1de954bf3ee05247cdff0c43d71dc9535803d/flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/sink/KafkaCommitterTest.java#L43
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @TempDir File outDir;
+
+ public static final MiniClusterExtension MINI_CLUSTER_RESOURCE =
+ new MiniClusterExtension(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ @RegisterExtension
+ public static AllCallbackWrapper<MiniClusterExtension> allCallbackWrapper =
+ new AllCallbackWrapper<>(MINI_CLUSTER_RESOURCE);
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
+ public void testBoundedTextFileSourceWithJackson() throws Exception {
+ writeFile(outDir, "data.csv", CSV_LINES);
+
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(outDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
Review comment:
Is this needed?
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/TableCsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.toLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class TableCsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
+ public void testProjectPushDown() throws Exception {
Review comment:
I thought projection push down is not supported. Why do you have a test for it?
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @TempDir File outDir;
+
+ public static final MiniClusterExtension MINI_CLUSTER_RESOURCE =
+ new MiniClusterExtension(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ @RegisterExtension
+ public static AllCallbackWrapper<MiniClusterExtension> allCallbackWrapper =
+ new AllCallbackWrapper<>(MINI_CLUSTER_RESOURCE);
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
+ public void testBoundedTextFileSourceWithJackson() throws Exception {
+ writeFile(outDir, "data.csv", CSV_LINES);
+
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(outDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
+ // fromCollection is not bounded, using fromSequence instead
+ final List<CityPojo> pojosList = Arrays.asList(pojos); // needs to be Serializable
+ final DataStream<Integer> sequence =
+ env.fromSequence(0, pojos.length - 1).map(Long::intValue);
+ final DataStream<CityPojo> stream = sequence.map(pojosList::get).returns(CityPojo.class);
+
+ FileSink<CityPojo> sink =
+ FileSink.forBulkFormat(new Path(outDir.toURI()), factoryForPojo(CityPojo.class))
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+
+ stream.sinkTo(sink);
+ env.execute();
+
+ String[] result = getResultsFromSinkFiles(outDir);
+
+ assertThat(result).containsExactlyInAnyOrder(CSV_LINES);
+ }
+
+ @NotNull
+ private String[] getResultsFromSinkFiles(File outDir) throws IOException {
+ final Map<File, String> contents = getFileContentByPath(outDir);
+
+ List<String> resultList =
+ contents.entrySet().stream()
+ .flatMap(e -> Arrays.stream(e.getValue().split("\n")))
+ .collect(Collectors.toList());
+
+ String[] result = resultList.toArray(new String[0]);
+ return result;
+ }
+
+ private static <T> BulkWriter.Factory<T> factoryForPojo(Class<T> pojoClass) {
+ final Converter<T, T, Void> converter = (value, context) -> value;
+ final CsvMapper csvMapper = new CsvMapper();
+ final CsvSchema schema = csvMapper.schemaFor(pojoClass).withoutQuoteChar();
+ return (out) -> new CsvBulkWriter<>(csvMapper, schema, converter, null, out);
+ }
+
+ private static Map<File, String> getFileContentByPath(File directory) throws IOException {
+ Map<File, String> contents = new HashMap<>(4);
+
+ final Collection<File> filesInBucket = FileUtils.listFiles(directory, null, true);
+ for (File file : filesInBucket) {
+ contents.put(file, FileUtils.readFileToString(file));
+ }
+ return contents;
+ }
+
+ @JsonPropertyOrder({
+ "city",
+ "lat",
+ "lng",
+ "country",
+ "iso2",
+ "adminName",
+ "capital",
+ "population"
+ })
+ public static class CityPojo implements Serializable {
+ public String city;
+ public BigDecimal lat;
+ public BigDecimal lng;
+ public String country;
+ public String iso2;
+ public String adminName;
+ public String capital;
+ public long population;
+
+ public CityPojo() {};
+
+ public CityPojo(
+ String city,
+ BigDecimal lat,
+ BigDecimal lng,
+ String country,
+ String iso2,
+ String admin_name,
+ String capital,
+ long population) {
+ this.city = city;
+ this.lat = lat;
+ this.lng = lng;
+ this.country = country;
+ this.iso2 = iso2;
+ this.adminName = admin_name;
+ this.capital = capital;
+ this.population = population;
+ }
+
+ @Override
+ public String toString() {
+ return "CitiesPojo{"
+ + "city='"
+ + city
+ + '\''
+ + ", lat="
+ + lat
+ + ", lng="
+ + lng
+ + ", country='"
+ + country
+ + '\''
+ + ", iso2='"
+ + iso2
+ + '\''
+ + ", adminName='"
+ + adminName
+ + '\''
+ + ", capital='"
+ + capital
+ + '\''
+ + ", population="
+ + population
+ + '}';
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ CityPojo that = (CityPojo) o;
+ return population == that.population
+ && Objects.equals(city, that.city)
+ && Objects.equals(lat, that.lat)
+ && Objects.equals(lng, that.lng)
+ && Objects.equals(country, that.country)
+ && Objects.equals(iso2, that.iso2)
+ && Objects.equals(adminName, that.adminName)
+ && Objects.equals(capital, that.capital);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(city, lat, lng, country, iso2, adminName, capital, population);
+ }
+ }
+
+ private static <T> List<T> initializeSourceAndReadData(File testDir, StreamFormat<T> csvFormat)
+ throws Exception {
+ final FileSource<T> source =
+ FileSource.forRecordStreamFormat(csvFormat, Path.fromLocalFile(testDir)).build();
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
+ final DataStream<T> stream =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "file-source");
+
+ return getResultsFromStream(stream);
+ }
+
+ @NotNull
+ private static <T> List<T> getResultsFromStream(DataStream<T> stream) throws Exception {
+ final ClientAndIterator<T> client =
+ DataStreamUtils.collectWithClient(stream, "Bounded Results Fetch");
+
+ final List<T> result = new ArrayList<>();
+ while (client.iterator.hasNext()) {
+ T next = client.iterator.next();
+ result.add(next);
+ }
+ return result;
+ }
+
+ // ------------------------------------------------------------------------
+ // test data
+ // ------------------------------------------------------------------------
+
+ private static final String[] CSV_LINES =
Review comment:
Can you move all static variables to the beginning of the class? The variable declaration should be always at the top of the file that other developers do not have to search them first. You can still generate the test data in a method and put it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
* f883156429a6510b06a05e58ddd77fb8e0554516 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181)
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
* f883156429a6510b06a05e58ddd77fb8e0554516 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r750166974
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/reader/DelimitedFormat.java
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.src.reader;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.FSDataInputStream;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Scanner;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A reader format that reads blocks of bytes from a file. The blocks are separated by the specified
+ * delimiting pattern and deserialized using the provided deserialization schema.
+ *
+ * <p>The reader uses Java's built-in {@link Scanner} to decode the byte stream using various
+ * supported charset encodings.
+ *
+ * <p>This format does not support optimized recovery from checkpoints. On recovery, it will re-read
+ * and discard the number of lined that were processed before the last checkpoint. That is due to
+ * the fact that the offsets of lines in the file cannot be tracked through the charset decoders
+ * with their internal buffering of stream input and charset decoder state.
+ */
+@PublicEvolving
+public class DelimitedFormat<T> extends SimpleStreamFormat<T> {
Review comment:
This is an obsolete approach, please ignore. This PR is still WIP and not ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0e3bc85342e7ae33117597931baa83f7c78c19a4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r790581794
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectWriter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** A simple {@link BulkWriter} implementation based on Jackson CSV transformations. */
+class CsvBulkWriter<T, R, C> implements BulkWriter<T> {
+
+ private final FSDataOutputStream stream;
+ @Nullable private final Converter<T, R, C> converter;
+ private final C converterContext;
Review comment:
Oops.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r789599660
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @TempDir File outDir;
+
+ public static final MiniClusterExtension MINI_CLUSTER_RESOURCE =
+ new MiniClusterExtension(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ @RegisterExtension
+ public static AllCallbackWrapper<MiniClusterExtension> allCallbackWrapper =
+ new AllCallbackWrapper<>(MINI_CLUSTER_RESOURCE);
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
+ public void testBoundedTextFileSourceWithJackson() throws Exception {
+ writeFile(outDir, "data.csv", CSV_LINES);
+
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(outDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
+ // fromCollection is not bounded, using fromSequence instead
+ final List<CityPojo> pojosList = Arrays.asList(pojos); // needs to be Serializable
+ final DataStream<Integer> sequence =
+ env.fromSequence(0, pojos.length - 1).map(Long::intValue);
+ final DataStream<CityPojo> stream = sequence.map(pojosList::get).returns(CityPojo.class);
+
+ FileSink<CityPojo> sink =
+ FileSink.forBulkFormat(new Path(outDir.toURI()), factoryForPojo(CityPojo.class))
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+
+ stream.sinkTo(sink);
+ env.execute();
+
+ String[] result = getResultsFromSinkFiles(outDir);
+
+ assertThat(result).containsExactlyInAnyOrder(CSV_LINES);
+ }
+
+ @NotNull
+ private String[] getResultsFromSinkFiles(File outDir) throws IOException {
+ final Map<File, String> contents = getFileContentByPath(outDir);
+
+ List<String> resultList =
+ contents.entrySet().stream()
+ .flatMap(e -> Arrays.stream(e.getValue().split("\n")))
+ .collect(Collectors.toList());
+
+ String[] result = resultList.toArray(new String[0]);
+ return result;
+ }
+
+ private static <T> BulkWriter.Factory<T> factoryForPojo(Class<T> pojoClass) {
+ final Converter<T, T, Void> converter = (value, context) -> value;
+ final CsvMapper csvMapper = new CsvMapper();
+ final CsvSchema schema = csvMapper.schemaFor(pojoClass).withoutQuoteChar();
+ return (out) -> new CsvBulkWriter<>(csvMapper, schema, converter, null, out);
+ }
+
+ private static Map<File, String> getFileContentByPath(File directory) throws IOException {
+ Map<File, String> contents = new HashMap<>(4);
+
+ final Collection<File> filesInBucket = FileUtils.listFiles(directory, null, true);
+ for (File file : filesInBucket) {
+ contents.put(file, FileUtils.readFileToString(file));
+ }
+ return contents;
+ }
+
+ @JsonPropertyOrder({
+ "city",
+ "lat",
+ "lng",
+ "country",
+ "iso2",
+ "adminName",
+ "capital",
+ "population"
+ })
+ public static class CityPojo implements Serializable {
+ public String city;
+ public BigDecimal lat;
+ public BigDecimal lng;
+ public String country;
+ public String iso2;
+ public String adminName;
+ public String capital;
+ public long population;
+
+ public CityPojo() {};
+
+ public CityPojo(
+ String city,
+ BigDecimal lat,
+ BigDecimal lng,
+ String country,
+ String iso2,
+ String admin_name,
+ String capital,
+ long population) {
+ this.city = city;
+ this.lat = lat;
+ this.lng = lng;
+ this.country = country;
+ this.iso2 = iso2;
+ this.adminName = admin_name;
+ this.capital = capital;
+ this.population = population;
+ }
+
+ @Override
+ public String toString() {
+ return "CitiesPojo{"
+ + "city='"
+ + city
+ + '\''
+ + ", lat="
+ + lat
+ + ", lng="
+ + lng
+ + ", country='"
+ + country
+ + '\''
+ + ", iso2='"
+ + iso2
+ + '\''
+ + ", adminName='"
+ + adminName
+ + '\''
+ + ", capital='"
+ + capital
+ + '\''
+ + ", population="
+ + population
+ + '}';
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ CityPojo that = (CityPojo) o;
+ return population == that.population
+ && Objects.equals(city, that.city)
+ && Objects.equals(lat, that.lat)
+ && Objects.equals(lng, that.lng)
+ && Objects.equals(country, that.country)
+ && Objects.equals(iso2, that.iso2)
+ && Objects.equals(adminName, that.adminName)
+ && Objects.equals(capital, that.capital);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(city, lat, lng, country, iso2, adminName, capital, population);
+ }
+ }
+
+ private static <T> List<T> initializeSourceAndReadData(File testDir, StreamFormat<T> csvFormat)
+ throws Exception {
+ final FileSource<T> source =
+ FileSource.forRecordStreamFormat(csvFormat, Path.fromLocalFile(testDir)).build();
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
+ final DataStream<T> stream =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "file-source");
+
+ return getResultsFromStream(stream);
+ }
+
+ @NotNull
+ private static <T> List<T> getResultsFromStream(DataStream<T> stream) throws Exception {
+ final ClientAndIterator<T> client =
+ DataStreamUtils.collectWithClient(stream, "Bounded Results Fetch");
+
+ final List<T> result = new ArrayList<>();
+ while (client.iterator.hasNext()) {
+ T next = client.iterator.next();
+ result.add(next);
+ }
+ return result;
+ }
+
+ // ------------------------------------------------------------------------
+ // test data
+ // ------------------------------------------------------------------------
+
+ private static final String[] CSV_LINES =
Review comment:
No worries I meant something like
```java
private static final String[] CSV_LINES = getCsvLines();
```
to prevent having a lot of lines for the initialization just for next time :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* e089768e55137235bd5b85a0b6b97b00dd0d23f9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119)
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r789558120
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @TempDir File outDir;
+
+ public static final MiniClusterExtension MINI_CLUSTER_RESOURCE =
+ new MiniClusterExtension(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ @RegisterExtension
+ public static AllCallbackWrapper<MiniClusterExtension> allCallbackWrapper =
Review comment:
Got some wrong "inspirations"
https://github.com/apache/flink/blob/49a05f4fb94370dd04480c16b04732c8a854cbbb/flink-runtime/src/test/java/org/apache/flink/runtime/taskmanager/TaskCancelAsyncProducerConsumerITCase.java#L80
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02355cdca67d73500f5251834bc07df18ec1f2a8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981)
* 35eb6edee6ce7d07bca66782ed264bdb00fbdc57 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-1006656067
@fapaul thanks a lot for your feedback. I've addressed your comments, apart from the IT/Unit-test related question. Actually, I am bit confused with that regard - my implementation is based largely on these `FileSource` tests:
https://github.com/apache/flink/blob/575aa2dbfb25af65436d60dd8ed2c0b1d0aa2ef8/flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/src/FileSourceTextLinesITCase.java#L126
However, the tests still work as expected after removing the `MiniCluster` from the test completely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r792641970
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFileFormatFactory.java
##########
@@ -109,17 +110,21 @@ public CsvBulkDecodingFormat(ReadableConfig formatOptions) {
final RowType physicalRowType = (RowType) physicalDataType.getLogicalType();
final CsvSchema schema = buildCsvSchema(physicalRowType, formatOptions);
+ final boolean ignoreParseErrors =
+ formatOptions.getOptional(IGNORE_PARSE_ERRORS).isPresent();
final Converter<JsonNode, RowData, Void> converter =
(Converter)
- new CsvToRowDataConverters(false)
+ new CsvToRowDataConverters(ignoreParseErrors)
.createRowConverter(projectedRowType, true);
- return new StreamFormatAdapter<>(
+ CsvReaderFormat<RowData> csvReaderFormat =
new CsvReaderFormat<>(
new CsvMapper(),
schema,
JsonNode.class,
converter,
- context.createTypeInformation(projectedDataType)));
+ context.createTypeInformation(projectedDataType));
+ csvReaderFormat.setIgnoreParseErrors(ignoreParseErrors);
Review comment:
Nit: WDYT about passing the complete configuration to the `Reader` class? It feels more extensible in case we also need to pass other configurations to the `Reader`.
The setter, in this case, is only invoked one time and it is similar to a builder method.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void> converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(), null) : null;
+ while (true) {
Review comment:
Why do you now use a while loop?
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void> converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(), null) : null;
+ while (true) {
+ try {
+ if (iterator.hasNext()) {
+ R nextElement = iterator.next();
+ return converter.convert(nextElement, null);
+ } else {
+ return null;
+ }
+ } catch (Throwable t) {
Review comment:
Is it necessary to catch all kinds of throwables here or is `IOException` enough? It can be dangerous to catch all kinds of exceptions i.e. if you handle the `InterruptedException` you also want to set the interrupted flag of the Thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
* f883156429a6510b06a05e58ddd77fb8e0554516 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r792676507
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void> converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(), null) : null;
+ while (true) {
+ try {
+ if (iterator.hasNext()) {
+ R nextElement = iterator.next();
+ return converter.convert(nextElement, null);
+ } else {
+ return null;
+ }
+ } catch (Throwable t) {
Review comment:
I decided not to reinvent the wheel and handled it exactly in the same way as this option is currently handled in the existing implementation to avoid surprises for the users that already rely on this functionality:
https://github.com/apache/flink/blob/16452fbb57c11418796912251152947601572106/flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvRowDataDeserializationSchema.java#L157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
* f883156429a6510b06a05e58ddd77fb8e0554516 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r792672907
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void> converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(), null) : null;
+ while (true) {
Review comment:
An arbitrary number of lines can be malformed. Let's say this is a single CSV file
```
a,b,c // header
1,2,3 // expected
1, // malformed
1,2,3,4 // malformed
5,6,7 // expected
```
Should produce [1,2,3] and [5,6,7] and ignore the rest. The number of malformed elements is unknown.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void> converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(), null) : null;
+ while (true) {
Review comment:
An arbitrary number of lines can be malformed. Let's say this is a single CSV file
```
a,b,c // header
1,2,3 // expected
1, // malformed
1,2,3,4 // malformed
5,6,7 // expected
```
Should produce [1,2,3] and [5,6,7] and ignore the rest. The number of malformed elements is unknown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r792672907
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void> converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(), null) : null;
+ while (true) {
Review comment:
An arbitrary number of lines can be malformed. Let's say this is a single CSV file
```
a,b,c // header
1,2,3 // expected
1, // malformed
1,2,3,4 // malformed
5,6,7 // expected
```
Should produce [1,2,3] and [5,6,7] and ignore the rest. The number of malformed elements is unknown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r797813488
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
@slinkydeveloper as discussed, https://issues.apache.org/jira/browse/FLINK-25931.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
slinkydeveloper commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r768837234
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/utils/JsonPlanTestBase.java
##########
@@ -92,17 +92,27 @@ protected void createTestValuesSourceTable(
@Nullable String partitionFields,
Map<String, String> extraProperties) {
checkArgument(fieldNameAndTypes.length > 0);
- String partitionedBy =
- StringUtils.isNullOrWhitespaceOnly(partitionFields)
- ? ""
- : "\n partitioned by (" + partitionFields + ") \n";
+
String dataId = TestValuesTableFactory.registerData(data);
Map<String, String> properties = new HashMap<>();
properties.put("connector", "values");
properties.put("data-id", dataId);
properties.put("bounded", "true");
properties.put("disable-lookup", "true");
properties.putAll(extraProperties);
+ createTestSourceTable(tableName, fieldNameAndTypes, partitionFields, properties);
+ }
+
+ protected void createTestSourceTable(
+ String tableName,
+ String[] fieldNameAndTypes,
+ @Nullable String partitionFields,
+ Map<String, String> properties) {
+ checkArgument(fieldNameAndTypes.length > 0);
+ String partitionedBy =
+ StringUtils.isNullOrWhitespaceOnly(partitionFields)
+ ? ""
+ : "\n partitioned by (" + partitionFields + ") \n";
Review comment:
please remove this function usage and rewrite it using `Stream`s instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770157114
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##########
@@ -125,128 +70,88 @@ public String factoryIdentifier() {
@Override
public Set<ConfigOption<?>> optionalOptions() {
- Set<ConfigOption<?>> options = new HashSet<>();
- options.add(FIELD_DELIMITER);
- options.add(DISABLE_QUOTE_CHARACTER);
- options.add(QUOTE_CHARACTER);
- options.add(ALLOW_COMMENTS);
- options.add(IGNORE_PARSE_ERRORS);
- options.add(ARRAY_ELEMENT_DELIMITER);
- options.add(ESCAPE_CHARACTER);
- options.add(NULL_LITERAL);
- return options;
- }
-
- // ------------------------------------------------------------------------
- // Validation
- // ------------------------------------------------------------------------
-
- static void validateFormatOptions(ReadableConfig tableOptions) {
- final boolean hasQuoteCharacter = tableOptions.getOptional(QUOTE_CHARACTER).isPresent();
- final boolean isDisabledQuoteCharacter = tableOptions.get(DISABLE_QUOTE_CHARACTER);
- if (isDisabledQuoteCharacter && hasQuoteCharacter) {
- throw new ValidationException(
- "Format cannot define a quote character and disabled quote character at the same time.");
- }
- // Validate the option value must be a single char.
- validateCharacterVal(tableOptions, FIELD_DELIMITER, true);
- validateCharacterVal(tableOptions, ARRAY_ELEMENT_DELIMITER);
- validateCharacterVal(tableOptions, QUOTE_CHARACTER);
- validateCharacterVal(tableOptions, ESCAPE_CHARACTER);
+ return CsvCommons.optionalOptions();
}
- /** Validates the option {@code option} value must be a Character. */
- private static void validateCharacterVal(
- ReadableConfig tableOptions, ConfigOption<String> option) {
- validateCharacterVal(tableOptions, option, false);
- }
+ @Override
+ public BulkDecodingFormat<RowData> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
- /**
- * Validates the option {@code option} value must be a Character.
- *
- * @param tableOptions the table options
- * @param option the config option
- * @param unescape whether to unescape the option value
- */
- private static void validateCharacterVal(
- ReadableConfig tableOptions, ConfigOption<String> option, boolean unescape) {
- if (tableOptions.getOptional(option).isPresent()) {
- final String value =
- unescape
- ? StringEscapeUtils.unescapeJava(tableOptions.get(option))
- : tableOptions.get(option);
- if (value.length() != 1) {
- throw new ValidationException(
- String.format(
- "Option '%s.%s' must be a string with single character, but was: %s",
- IDENTIFIER, option.key(), tableOptions.get(option)));
- }
- }
+ return new CsvBulkDecodingFormat(formatOptions);
}
- // ------------------------------------------------------------------------
- // Utilities
- // ------------------------------------------------------------------------
+ private static class CsvBulkDecodingFormat
+ implements BulkDecodingFormat<RowData>,
+ ProjectableDecodingFormat<BulkFormat<RowData, FileSourceSplit>> {
- private static void configureDeserializationSchema(
- ReadableConfig formatOptions, CsvRowDataDeserializationSchema.Builder schemaBuilder) {
- formatOptions
- .getOptional(FIELD_DELIMITER)
- .map(delimiter -> StringEscapeUtils.unescapeJava(delimiter).charAt(0))
- .ifPresent(schemaBuilder::setFieldDelimiter);
+ private final ReadableConfig formatOptions;
- if (formatOptions.get(DISABLE_QUOTE_CHARACTER)) {
- schemaBuilder.disableQuoteCharacter();
- } else {
- formatOptions
- .getOptional(QUOTE_CHARACTER)
- .map(quote -> quote.charAt(0))
- .ifPresent(schemaBuilder::setQuoteCharacter);
+ public CsvBulkDecodingFormat(ReadableConfig formatOptions) {
+ this.formatOptions = formatOptions;
}
- formatOptions.getOptional(ALLOW_COMMENTS).ifPresent(schemaBuilder::setAllowComments);
-
- formatOptions
- .getOptional(IGNORE_PARSE_ERRORS)
- .ifPresent(schemaBuilder::setIgnoreParseErrors);
-
- formatOptions
- .getOptional(ARRAY_ELEMENT_DELIMITER)
- .ifPresent(schemaBuilder::setArrayElementDelimiter);
-
- formatOptions
- .getOptional(ESCAPE_CHARACTER)
- .map(escape -> escape.charAt(0))
- .ifPresent(schemaBuilder::setEscapeCharacter);
+ @Override
+ // TODO: is it possible to avoid the cast with a reasonable effort?
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ public BulkFormat<RowData, FileSourceSplit> createRuntimeDecoder(
+ Context context, DataType physicalDataType, int[][] projections) {
+
+ final DataType projectedDataDype = Projection.of(projections).project(physicalDataType);
+ final RowType rowTypeProjected = (RowType) projectedDataDype.getLogicalType();
+
+ final RowType rowType = (RowType) physicalDataType.getLogicalType();
Review comment:
How about the `getLogicalType()` part? Would `logicalRowType` be more appropriate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770169936
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.formats.csv.RowDataToCsvConverters.RowDataToCsvConverter.RowDataToCsvFormatConverterContext;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectWriter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import java.io.IOException;
+
+/** A simple {@link BulkWriter} implementation based on Jackson CSV transformations. */
+@PublicEvolving
+public class CsvBulkWriter<T> implements BulkWriter<T> {
+
+ private final FSDataOutputStream stream;
+ private final CsvMapper mapper;
+ private final Converter<T, JsonNode, RowDataToCsvFormatConverterContext> converter;
+ private final ObjectWriter csvWriter;
+ /** Reusable within the converter. * */
+ private final transient ObjectNode container;
Review comment:
This is in accordance with the design choice explicitly made in a similar situation in the recent addition:
https://github.com/apache/flink/commit/b1e436a109c6473b794f02ec0a853d1ae6df6c83#diff-58a105b8d92eadfb0af487ba6b2bce5a00955fceb2077e6b7f2d1fa01592a50bR84
One could argue that as it is a throw-away object that does not itself contain any configuration there is no strict necessity to ship it to the executors. Let me know what you think and if there are general guidelines for similar cases in Flink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "6f95051fa8153ff7745b479e75e44e9b0ea496f0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6f95051fa8153ff7745b479e75e44e9b0ea496f0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 29e6025b084eb1c850ab3901c757dcfcb442c67e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289)
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
* 6f95051fa8153ff7745b479e75e44e9b0ea496f0 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r771372489
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
Is this fully supported in Table-land? If so, then I agree we should change it. It's a good performance-improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 29e6025b084eb1c850ab3901c757dcfcb442c67e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289)
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7e097463582d225c33b56c3d2f067a3954867a11 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599)
* 077423c8c94f59aade26c2c57001a4551a1b28af Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 077423c8c94f59aade26c2c57001a4551a1b28af Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971)
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 744b699a82b4640652050eb75eecaf78b2f10c2c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 744b699a82b4640652050eb75eecaf78b2f10c2c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105)
* e089768e55137235bd5b85a0b6b97b00dd0d23f9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770157114
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##########
@@ -125,128 +70,88 @@ public String factoryIdentifier() {
@Override
public Set<ConfigOption<?>> optionalOptions() {
- Set<ConfigOption<?>> options = new HashSet<>();
- options.add(FIELD_DELIMITER);
- options.add(DISABLE_QUOTE_CHARACTER);
- options.add(QUOTE_CHARACTER);
- options.add(ALLOW_COMMENTS);
- options.add(IGNORE_PARSE_ERRORS);
- options.add(ARRAY_ELEMENT_DELIMITER);
- options.add(ESCAPE_CHARACTER);
- options.add(NULL_LITERAL);
- return options;
- }
-
- // ------------------------------------------------------------------------
- // Validation
- // ------------------------------------------------------------------------
-
- static void validateFormatOptions(ReadableConfig tableOptions) {
- final boolean hasQuoteCharacter = tableOptions.getOptional(QUOTE_CHARACTER).isPresent();
- final boolean isDisabledQuoteCharacter = tableOptions.get(DISABLE_QUOTE_CHARACTER);
- if (isDisabledQuoteCharacter && hasQuoteCharacter) {
- throw new ValidationException(
- "Format cannot define a quote character and disabled quote character at the same time.");
- }
- // Validate the option value must be a single char.
- validateCharacterVal(tableOptions, FIELD_DELIMITER, true);
- validateCharacterVal(tableOptions, ARRAY_ELEMENT_DELIMITER);
- validateCharacterVal(tableOptions, QUOTE_CHARACTER);
- validateCharacterVal(tableOptions, ESCAPE_CHARACTER);
+ return CsvCommons.optionalOptions();
}
- /** Validates the option {@code option} value must be a Character. */
- private static void validateCharacterVal(
- ReadableConfig tableOptions, ConfigOption<String> option) {
- validateCharacterVal(tableOptions, option, false);
- }
+ @Override
+ public BulkDecodingFormat<RowData> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
- /**
- * Validates the option {@code option} value must be a Character.
- *
- * @param tableOptions the table options
- * @param option the config option
- * @param unescape whether to unescape the option value
- */
- private static void validateCharacterVal(
- ReadableConfig tableOptions, ConfigOption<String> option, boolean unescape) {
- if (tableOptions.getOptional(option).isPresent()) {
- final String value =
- unescape
- ? StringEscapeUtils.unescapeJava(tableOptions.get(option))
- : tableOptions.get(option);
- if (value.length() != 1) {
- throw new ValidationException(
- String.format(
- "Option '%s.%s' must be a string with single character, but was: %s",
- IDENTIFIER, option.key(), tableOptions.get(option)));
- }
- }
+ return new CsvBulkDecodingFormat(formatOptions);
}
- // ------------------------------------------------------------------------
- // Utilities
- // ------------------------------------------------------------------------
+ private static class CsvBulkDecodingFormat
+ implements BulkDecodingFormat<RowData>,
+ ProjectableDecodingFormat<BulkFormat<RowData, FileSourceSplit>> {
- private static void configureDeserializationSchema(
- ReadableConfig formatOptions, CsvRowDataDeserializationSchema.Builder schemaBuilder) {
- formatOptions
- .getOptional(FIELD_DELIMITER)
- .map(delimiter -> StringEscapeUtils.unescapeJava(delimiter).charAt(0))
- .ifPresent(schemaBuilder::setFieldDelimiter);
+ private final ReadableConfig formatOptions;
- if (formatOptions.get(DISABLE_QUOTE_CHARACTER)) {
- schemaBuilder.disableQuoteCharacter();
- } else {
- formatOptions
- .getOptional(QUOTE_CHARACTER)
- .map(quote -> quote.charAt(0))
- .ifPresent(schemaBuilder::setQuoteCharacter);
+ public CsvBulkDecodingFormat(ReadableConfig formatOptions) {
+ this.formatOptions = formatOptions;
}
- formatOptions.getOptional(ALLOW_COMMENTS).ifPresent(schemaBuilder::setAllowComments);
-
- formatOptions
- .getOptional(IGNORE_PARSE_ERRORS)
- .ifPresent(schemaBuilder::setIgnoreParseErrors);
-
- formatOptions
- .getOptional(ARRAY_ELEMENT_DELIMITER)
- .ifPresent(schemaBuilder::setArrayElementDelimiter);
-
- formatOptions
- .getOptional(ESCAPE_CHARACTER)
- .map(escape -> escape.charAt(0))
- .ifPresent(schemaBuilder::setEscapeCharacter);
+ @Override
+ // TODO: is it possible to avoid the cast with a reasonable effort?
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ public BulkFormat<RowData, FileSourceSplit> createRuntimeDecoder(
+ Context context, DataType physicalDataType, int[][] projections) {
+
+ final DataType projectedDataDype = Projection.of(projections).project(physicalDataType);
+ final RowType rowTypeProjected = (RowType) projectedDataDype.getLogicalType();
+
+ final RowType rowType = (RowType) physicalDataType.getLogicalType();
Review comment:
How about `getLogicalType()`? Would `logicalRowType` be more appropriate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
slinkydeveloper commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770301625
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##########
@@ -125,128 +70,88 @@ public String factoryIdentifier() {
@Override
public Set<ConfigOption<?>> optionalOptions() {
- Set<ConfigOption<?>> options = new HashSet<>();
- options.add(FIELD_DELIMITER);
- options.add(DISABLE_QUOTE_CHARACTER);
- options.add(QUOTE_CHARACTER);
- options.add(ALLOW_COMMENTS);
- options.add(IGNORE_PARSE_ERRORS);
- options.add(ARRAY_ELEMENT_DELIMITER);
- options.add(ESCAPE_CHARACTER);
- options.add(NULL_LITERAL);
- return options;
- }
-
- // ------------------------------------------------------------------------
- // Validation
- // ------------------------------------------------------------------------
-
- static void validateFormatOptions(ReadableConfig tableOptions) {
- final boolean hasQuoteCharacter = tableOptions.getOptional(QUOTE_CHARACTER).isPresent();
- final boolean isDisabledQuoteCharacter = tableOptions.get(DISABLE_QUOTE_CHARACTER);
- if (isDisabledQuoteCharacter && hasQuoteCharacter) {
- throw new ValidationException(
- "Format cannot define a quote character and disabled quote character at the same time.");
- }
- // Validate the option value must be a single char.
- validateCharacterVal(tableOptions, FIELD_DELIMITER, true);
- validateCharacterVal(tableOptions, ARRAY_ELEMENT_DELIMITER);
- validateCharacterVal(tableOptions, QUOTE_CHARACTER);
- validateCharacterVal(tableOptions, ESCAPE_CHARACTER);
+ return CsvCommons.optionalOptions();
}
- /** Validates the option {@code option} value must be a Character. */
- private static void validateCharacterVal(
- ReadableConfig tableOptions, ConfigOption<String> option) {
- validateCharacterVal(tableOptions, option, false);
- }
+ @Override
+ public BulkDecodingFormat<RowData> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
- /**
- * Validates the option {@code option} value must be a Character.
- *
- * @param tableOptions the table options
- * @param option the config option
- * @param unescape whether to unescape the option value
- */
- private static void validateCharacterVal(
- ReadableConfig tableOptions, ConfigOption<String> option, boolean unescape) {
- if (tableOptions.getOptional(option).isPresent()) {
- final String value =
- unescape
- ? StringEscapeUtils.unescapeJava(tableOptions.get(option))
- : tableOptions.get(option);
- if (value.length() != 1) {
- throw new ValidationException(
- String.format(
- "Option '%s.%s' must be a string with single character, but was: %s",
- IDENTIFIER, option.key(), tableOptions.get(option)));
- }
- }
+ return new CsvBulkDecodingFormat(formatOptions);
}
- // ------------------------------------------------------------------------
- // Utilities
- // ------------------------------------------------------------------------
+ private static class CsvBulkDecodingFormat
+ implements BulkDecodingFormat<RowData>,
+ ProjectableDecodingFormat<BulkFormat<RowData, FileSourceSplit>> {
- private static void configureDeserializationSchema(
- ReadableConfig formatOptions, CsvRowDataDeserializationSchema.Builder schemaBuilder) {
- formatOptions
- .getOptional(FIELD_DELIMITER)
- .map(delimiter -> StringEscapeUtils.unescapeJava(delimiter).charAt(0))
- .ifPresent(schemaBuilder::setFieldDelimiter);
+ private final ReadableConfig formatOptions;
- if (formatOptions.get(DISABLE_QUOTE_CHARACTER)) {
- schemaBuilder.disableQuoteCharacter();
- } else {
- formatOptions
- .getOptional(QUOTE_CHARACTER)
- .map(quote -> quote.charAt(0))
- .ifPresent(schemaBuilder::setQuoteCharacter);
+ public CsvBulkDecodingFormat(ReadableConfig formatOptions) {
+ this.formatOptions = formatOptions;
}
- formatOptions.getOptional(ALLOW_COMMENTS).ifPresent(schemaBuilder::setAllowComments);
-
- formatOptions
- .getOptional(IGNORE_PARSE_ERRORS)
- .ifPresent(schemaBuilder::setIgnoreParseErrors);
-
- formatOptions
- .getOptional(ARRAY_ELEMENT_DELIMITER)
- .ifPresent(schemaBuilder::setArrayElementDelimiter);
-
- formatOptions
- .getOptional(ESCAPE_CHARACTER)
- .map(escape -> escape.charAt(0))
- .ifPresent(schemaBuilder::setEscapeCharacter);
+ @Override
+ // TODO: is it possible to avoid the cast with a reasonable effort?
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ public BulkFormat<RowData, FileSourceSplit> createRuntimeDecoder(
+ Context context, DataType physicalDataType, int[][] projections) {
+
+ final DataType projectedDataDype = Projection.of(projections).project(physicalDataType);
+ final RowType rowTypeProjected = (RowType) projectedDataDype.getLogicalType();
+
+ final RowType rowType = (RowType) physicalDataType.getLogicalType();
Review comment:
no, because `RowType` is a `LogicalType`. `physicalRowType` makes clear that this one is the physical type, and not the projected one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770679363
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##########
@@ -19,103 +19,48 @@
package org.apache.flink.formats.csv;
import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.common.serialization.SerializationSchema;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.configuration.ReadableConfig;
-import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.impl.StreamFormatAdapter;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.formats.common.Converter;
import org.apache.flink.table.connector.ChangelogMode;
-import org.apache.flink.table.connector.format.DecodingFormat;
-import org.apache.flink.table.connector.format.EncodingFormat;
-import org.apache.flink.table.connector.sink.DynamicTableSink;
-import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.Projection;
+import org.apache.flink.table.connector.format.BulkDecodingFormat;
+import org.apache.flink.table.connector.format.ProjectableDecodingFormat;
+import org.apache.flink.table.connector.source.DynamicTableSource.Context;
import org.apache.flink.table.data.RowData;
-import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.BulkReaderFormatFactory;
import org.apache.flink.table.factories.DynamicTableFactory;
-import org.apache.flink.table.factories.FactoryUtil;
-import org.apache.flink.table.factories.SerializationFormatFactory;
import org.apache.flink.table.types.DataType;
import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
import org.apache.commons.lang3.StringEscapeUtils;
import java.util.Collections;
-import java.util.HashSet;
import java.util.Set;
import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
-import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
-/**
- * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
- * and {@link DeserializationSchema}.
- */
+/** CSV format factory for file system. */
+//TODO: Rename back to CsvFileSystemFormatFactory? @Arvid
Review comment:
I had this thought because `BulkReader` is a `flink-connector-file` interface, and `CsvBulkWriter` writer into `FSDataOutputStream`. Both the encoder and the decoder exposed by the factory are for files.
I like the `TableCsvFormatFactory`, renamed.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##########
@@ -19,103 +19,48 @@
package org.apache.flink.formats.csv;
import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.common.serialization.SerializationSchema;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.configuration.ReadableConfig;
-import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.impl.StreamFormatAdapter;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.formats.common.Converter;
import org.apache.flink.table.connector.ChangelogMode;
-import org.apache.flink.table.connector.format.DecodingFormat;
-import org.apache.flink.table.connector.format.EncodingFormat;
-import org.apache.flink.table.connector.sink.DynamicTableSink;
-import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.Projection;
+import org.apache.flink.table.connector.format.BulkDecodingFormat;
+import org.apache.flink.table.connector.format.ProjectableDecodingFormat;
+import org.apache.flink.table.connector.source.DynamicTableSource.Context;
import org.apache.flink.table.data.RowData;
-import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.BulkReaderFormatFactory;
import org.apache.flink.table.factories.DynamicTableFactory;
-import org.apache.flink.table.factories.FactoryUtil;
-import org.apache.flink.table.factories.SerializationFormatFactory;
import org.apache.flink.table.types.DataType;
import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
import org.apache.commons.lang3.StringEscapeUtils;
import java.util.Collections;
-import java.util.HashSet;
import java.util.Set;
import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
-import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
-/**
- * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
- * and {@link DeserializationSchema}.
- */
+/** CSV format factory for file system. */
+//TODO: Rename back to CsvFileSystemFormatFactory? @Arvid
Review comment:
I had this thought because `BulkReader` is a `flink-connector-file` interface, and `CsvBulkWriter` writes into `FSDataOutputStream`. Both the encoder and the decoder exposed by the factory are for files.
I like the `TableCsvFormatFactory`, renamed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 29e6025b084eb1c850ab3901c757dcfcb442c67e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c1c04317c7fb3f4470cdf596756172105d25cda Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c1c04317c7fb3f4470cdf596756172105d25cda Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391)
* 7e097463582d225c33b56c3d2f067a3954867a11 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r771364334
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.unixTimestampToLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class CsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
+ public void testProjectPushDown() throws Exception {
Review comment:
> Do you mean testing the new CsvBulkWriter + CsvReaderFormat and not anything related to the ser/de schema-based (old) stuff?
Yes
> Another question - you point at the projection pushdown test, not sure if you meant it explicitly (new functionality?) or if it is a coincidence. Do we even have facilities for communicating pushdown from the streaming API?
I can only write comments on changed lines. So it's a coincidence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r773800347
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
@slinkydeveloper
I am not sure about this. The focus of this PR is
**Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.**
I would rather not touch the existing Ser/De schema-based functionality in this scope. Nothing will break if we won't support pushdown in the non-filesystem connectors for now and to me, it sounds like an improvement that should be done in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-1021629155
@fapaul
Done.
I've added two hotfix commits that I would like you to give a look at before I squash them.
1. Adds support for the missing `ignore-parse-errors` option. Some tests were failing because of that. It is a bit of a [special option](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/formats/csv/#csv-ignore-parse-errors) when compared to the parsing options of Jackson, that's why I decided to make it a setter not to "pollute" all of the factory methods with it.
2. Adds the `forwardOptions` from Francesco's PR. I believe that is all that is needed, but please confirm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
* f883156429a6510b06a05e58ddd77fb8e0554516 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
* f883156429a6510b06a05e58ddd77fb8e0554516 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181)
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-1022504990
@fapaul Done. I've also added an explicit test for the `withIgnoreParseErrors` option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
}, {
"hash" : "f51dee82fb528a7ae6f883c46be76b3e1b3af6c4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f51dee82fb528a7ae6f883c46be76b3e1b3af6c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183)
* f51dee82fb528a7ae6f883c46be76b3e1b3af6c4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ee8fa611b135924ad3a083ff71dbcbc2e1895351 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 077423c8c94f59aade26c2c57001a4551a1b28af Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971)
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "4b6e7dd98a6318a316647db354c5850f1969851b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4b6e7dd98a6318a316647db354c5850f1969851b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 744b699a82b4640652050eb75eecaf78b2f10c2c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105)
* 4b6e7dd98a6318a316647db354c5850f1969851b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 744b699a82b4640652050eb75eecaf78b2f10c2c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105)
* e089768e55137235bd5b85a0b6b97b00dd0d23f9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 744b699a82b4640652050eb75eecaf78b2f10c2c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r779590843
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.StreamSink;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy.build;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @ClassRule public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
+
+ @ClassRule
+ public static final MiniClusterWithClientResource miniClusterResource =
+ new MiniClusterWithClientResource(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
Review comment:
Ah, now I see what happened here :) It is actually correct to keep the mini cluster but I would suggest making the second test either in a separate test class because it is not an IT case or also converting it to an IT case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r789503305
##########
File path: flink-formats/flink-csv/pom.xml
##########
@@ -77,6 +77,14 @@ under the License.
<!-- test dependencies -->
+ <dependency>
Review comment:
Good point, it is a leftover from an earlier testing approach, not needed anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r789624039
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.StreamSink;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy.build;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @ClassRule public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
+
+ @ClassRule
+ public static final MiniClusterWithClientResource miniClusterResource =
+ new MiniClusterWithClientResource(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
Review comment:
I've added an explicit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r790515844
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,390 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLoggerExtension;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
Review comment:
Please only use assertJ
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvCommons.java
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.ValidationException;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/** A class with common CSV format constants and utility methods. */
+class CsvCommons {
+
+ public static final String IDENTIFIER = "csv";
+
+ // ------------------------------------------------------------------------
+ // Validation
+ // ------------------------------------------------------------------------
+
+ static void validateFormatOptions(ReadableConfig tableOptions) {
+ final boolean hasQuoteCharacter = tableOptions.getOptional(QUOTE_CHARACTER).isPresent();
+ final boolean isDisabledQuoteCharacter = tableOptions.get(DISABLE_QUOTE_CHARACTER);
+ if (isDisabledQuoteCharacter && hasQuoteCharacter) {
+ throw new ValidationException(
+ "Format cannot define a quote character and disabled quote character at the same time.");
+ }
+ // Validate the option value must be a single char.
+ validateCharacterVal(tableOptions, FIELD_DELIMITER, true);
+ validateCharacterVal(tableOptions, ARRAY_ELEMENT_DELIMITER);
+ validateCharacterVal(tableOptions, QUOTE_CHARACTER);
+ validateCharacterVal(tableOptions, ESCAPE_CHARACTER);
+ }
+
+ /** Validates the option {@code option} value must be a Character. */
+ private static void validateCharacterVal(
+ ReadableConfig tableOptions, ConfigOption<String> option) {
+ validateCharacterVal(tableOptions, option, false);
+ }
+
+ /**
+ * Validates the option {@code option} value must be a Character.
+ *
+ * @param tableOptions the table options
+ * @param option the config option
+ * @param unescape whether to unescape the option value
+ */
+ private static void validateCharacterVal(
+ ReadableConfig tableOptions, ConfigOption<String> option, boolean unescape) {
+ if (!tableOptions.getOptional(option).isPresent()) {
+ return;
+ }
+
+ final String value =
+ unescape
+ ? StringEscapeUtils.unescapeJava(tableOptions.get(option))
+ : tableOptions.get(option);
+ if (value.length() != 1) {
+ throw new ValidationException(
+ String.format(
+ "Option '%s.%s' must be a string with single character, but was: %s",
+ IDENTIFIER, option.key(), tableOptions.get(option)));
+ }
+ }
+
+ public static Set<ConfigOption<?>> optionalOptions() {
Review comment:
Can you align the options with this PR https://github.com/apache/flink/pull/18290 ?
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectWriter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** A simple {@link BulkWriter} implementation based on Jackson CSV transformations. */
+class CsvBulkWriter<T, R, C> implements BulkWriter<T> {
+
+ private final FSDataOutputStream stream;
+ @Nullable private final Converter<T, R, C> converter;
+ private final C converterContext;
Review comment:
I think the wrong variable is now annotated nullable.
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,390 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLoggerExtension;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+@ExtendWith({TestLoggerExtension.class})
+public class DataStreamCsvITCase {
+
+ private static final int PARALLELISM = 4;
+
+ @TempDir File outDir;
+
+ private static final MiniClusterExtension MINI_CLUSTER_RESOURCE =
+ new MiniClusterExtension(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ @RegisterExtension
+ private static AllCallbackWrapper<MiniClusterExtension> allCallbackWrapper =
+ new AllCallbackWrapper<>(MINI_CLUSTER_RESOURCE);
+
+ // ------------------------------------------------------------------------
+ // test data
+ // ------------------------------------------------------------------------
+
+ private static final String[] CSV_LINES =
+ new String[] {
+ "Berlin,52.5167,13.3833,Germany,DE,Berlin,primary,3644826",
+ "San Francisco,37.7562,-122.443,United States,US,California,,3592294",
+ "Beijing,39.905,116.3914,China,CN,Beijing,primary,19433000"
+ };
+
+ private static final String[] CSV_LINES_PIPE_SEPARATED =
+ new String[] {
+ "Berlin|52.5167|13.3833|Germany|DE|Berlin|primary|3644826",
+ "San Francisco|37.7562|-122.443|United States|US|California||3592294",
+ "Beijing|39.905|116.3914|China|CN|Beijing|primary|19433000"
+ };
+
+ static final CityPojo[] POJOS =
+ new CityPojo[] {
+ new CityPojo(
+ "Berlin",
+ new BigDecimal("52.5167"),
+ new BigDecimal("13.3833"),
+ "Germany",
+ "DE",
+ "Berlin",
+ "primary",
+ 3644826L),
+ new CityPojo(
+ "San Francisco",
+ new BigDecimal("37.7562"),
+ new BigDecimal("-122.443"),
+ "United States",
+ "US",
+ "California",
+ "",
+ 3592294L),
+ new CityPojo(
+ "Beijing",
+ new BigDecimal("39.905"),
+ new BigDecimal("116.3914"),
+ "China",
+ "CN",
+ "Beijing",
+ "primary",
+ 19433000L)
+ };
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
+ public void testCsvReaderFormatFromPojo() throws Exception {
+ writeFile(outDir, "data.csv", CSV_LINES);
+
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(outDir, csvFormat);
+
+ assertEquals(Arrays.asList(POJOS), result);
+ }
+
+ @Test
+ public void testCsvReaderFormatFromSchema() throws Exception {
+ writeFile(outDir, "data.csv", CSV_LINES_PIPE_SEPARATED);
+
+ CsvMapper mapper = new CsvMapper();
+ CsvSchema schema =
+ mapper.schemaFor(CityPojo.class).withoutQuoteChar().withColumnSeparator('|');
+
+ final CsvReaderFormat<CityPojo> csvFormat =
+ CsvReaderFormat.forSchema(mapper, schema, TypeInformation.of(CityPojo.class));
+ final List<CityPojo> result = initializeSourceAndReadData(outDir, csvFormat);
+ assertEquals(Arrays.asList(POJOS), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+
+ // fromCollection is not bounded, using fromSequence instead
+ final List<CityPojo> pojosList = Arrays.asList(POJOS); // needs to be Serializable
+ final DataStream<Integer> sequence =
+ env.fromSequence(0, POJOS.length - 1).map(Long::intValue);
+ final DataStream<CityPojo> stream = sequence.map(pojosList::get).returns(CityPojo.class);
+
+ FileSink<CityPojo> sink =
+ FileSink.forBulkFormat(new Path(outDir.toURI()), factoryForPojo(CityPojo.class))
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+
+ stream.sinkTo(sink);
+ env.execute();
+
+ String[] result = getResultsFromSinkFiles(outDir);
+
+ assertThat(result).containsExactlyInAnyOrder(CSV_LINES);
+ }
+
+ @NotNull
+ private String[] getResultsFromSinkFiles(File outDir) throws IOException {
+ final Map<File, String> contents = getFileContentByPath(outDir);
+
+ List<String> resultList =
+ contents.entrySet().stream()
+ .flatMap(e -> Arrays.stream(e.getValue().split("\n")))
+ .collect(Collectors.toList());
+
+ String[] result = resultList.toArray(new String[0]);
+ return result;
+ }
+
+ private static <T> BulkWriter.Factory<T> factoryForPojo(Class<T> pojoClass) {
+ final Converter<T, T, Void> converter = (value, context) -> value;
+ final CsvMapper csvMapper = new CsvMapper();
+ final CsvSchema schema = csvMapper.schemaFor(pojoClass).withoutQuoteChar();
+ return (out) -> new CsvBulkWriter<>(csvMapper, schema, converter, null, out);
+ }
+
+ private static Map<File, String> getFileContentByPath(File directory) throws IOException {
+ Map<File, String> contents = new HashMap<>(4);
+
+ final Collection<File> filesInBucket = FileUtils.listFiles(directory, null, true);
+ for (File file : filesInBucket) {
+ contents.put(file, FileUtils.readFileToString(file));
+ }
+ return contents;
+ }
+
+ @JsonPropertyOrder({
+ "city",
+ "lat",
+ "lng",
+ "country",
+ "iso2",
+ "adminName",
+ "capital",
+ "population"
+ })
+ static class CityPojo implements Serializable {
+ public String city;
+ public BigDecimal lat;
+ public BigDecimal lng;
+ public String country;
+ public String iso2;
+ public String adminName;
+ public String capital;
+ public long population;
+
+ public CityPojo() {}
+
+ public CityPojo(
+ String city,
+ BigDecimal lat,
+ BigDecimal lng,
+ String country,
+ String iso2,
+ String adminName,
+ String capital,
+ long population) {
+ this.city = city;
+ this.lat = lat;
+ this.lng = lng;
+ this.country = country;
+ this.iso2 = iso2;
+ this.adminName = adminName;
+ this.capital = capital;
+ this.population = population;
+ }
+
+ @Override
+ public String toString() {
+ return "CitiesPojo{"
+ + "city='"
+ + city
+ + '\''
+ + ", lat="
+ + lat
+ + ", lng="
+ + lng
+ + ", country='"
+ + country
+ + '\''
+ + ", iso2='"
+ + iso2
+ + '\''
+ + ", adminName='"
+ + adminName
+ + '\''
+ + ", capital='"
+ + capital
+ + '\''
+ + ", population="
+ + population
+ + '}';
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ CityPojo that = (CityPojo) o;
+ return population == that.population
+ && Objects.equals(city, that.city)
+ && Objects.equals(lat, that.lat)
+ && Objects.equals(lng, that.lng)
+ && Objects.equals(country, that.country)
+ && Objects.equals(iso2, that.iso2)
+ && Objects.equals(adminName, that.adminName)
+ && Objects.equals(capital, that.capital);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(city, lat, lng, country, iso2, adminName, capital, population);
+ }
+ }
+
+ private static <T> List<T> initializeSourceAndReadData(File testDir, StreamFormat<T> csvFormat)
+ throws Exception {
+ final FileSource<T> source =
+ FileSource.forRecordStreamFormat(csvFormat, Path.fromLocalFile(testDir)).build();
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
+ final DataStream<T> stream =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "file-source");
+
+ return getResultsFromStream(stream);
+ }
+
+ @NotNull
+ private static <T> List<T> getResultsFromStream(DataStream<T> stream) throws Exception {
+ final ClientAndIterator<T> client =
+ DataStreamUtils.collectWithClient(stream, "Bounded Results Fetch");
+
+ final List<T> result = new ArrayList<>();
+ while (client.iterator.hasNext()) {
+ T next = client.iterator.next();
+ result.add(next);
+ }
+ return result;
+ }
+
+ // ------------------------------------------------------------------------
+ // Write data utils
+ // ------------------------------------------------------------------------
+
+ private static void writeFile(File testDir, String fileName, String[] lines)
+ throws IOException {
+ final File file = new File(testDir, fileName);
+ writeFileAtomically(file, lines);
+ }
+
+ private static void writeFileAtomically(final File file, final String[] lines)
+ throws IOException {
+ writeFileAtomically(file, lines, (v) -> v);
+ }
+
+ private static void writeFileAtomically(
+ final File file,
+ final String[] lines,
+ final FunctionWithException<OutputStream, OutputStream, IOException>
+ streamEncoderFactory)
+ throws IOException {
+
+ final File stagingFile =
+ new File(file.getParentFile(), ".tmp-" + UUID.randomUUID().toString());
+
+ try (final FileOutputStream fileOut = new FileOutputStream(stagingFile);
+ final OutputStream out = streamEncoderFactory.apply(fileOut);
+ final OutputStreamWriter encoder =
+ new OutputStreamWriter(out, StandardCharsets.UTF_8);
+ final PrintWriter writer = new PrintWriter(encoder)) {
+
+ for (String line : lines) {
+ writer.println(line);
+ }
+ }
+
+ final File parent = file.getParentFile();
+ assertTrue(parent.mkdirs() || parent.exists());
+
+ assertTrue(stagingFile.renameTo(file));
Review comment:
Please use assertj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-tbd][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9aa8115b9b3fa8cc853f7821c29f939870c79e5d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ee8fa611b135924ad3a083ff71dbcbc2e1895351 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635)
* 0e3bc85342e7ae33117597931baa83f7c78c19a4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on a change in pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
slinkydeveloper commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r750133859
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/reader/DelimitedFormat.java
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.src.reader;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.FSDataInputStream;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Scanner;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A reader format that reads blocks of bytes from a file. The blocks are separated by the specified
+ * delimiting pattern and deserialized using the provided deserialization schema.
+ *
+ * <p>The reader uses Java's built-in {@link Scanner} to decode the byte stream using various
+ * supported charset encodings.
+ *
+ * <p>This format does not support optimized recovery from checkpoints. On recovery, it will re-read
+ * and discard the number of lined that were processed before the last checkpoint. That is due to
+ * the fact that the offsets of lines in the file cannot be tracked through the charset decoders
+ * with their internal buffering of stream input and charset decoder state.
+ */
+@PublicEvolving
+public class DelimitedFormat<T> extends SimpleStreamFormat<T> {
Review comment:
I'm wondering, what is the difference between this format and `DeserializationSchemaAdapter`? Could the latter be dropped in favour of this new class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r778949745
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.StreamSink;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy.build;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @ClassRule public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
+
+ @ClassRule
+ public static final MiniClusterWithClientResource miniClusterResource =
+ new MiniClusterWithClientResource(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
Review comment:
If you want to write an IT case you can use the new File. I hope with 1.15 we can deprecate `StreamingFileSink`.
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.StreamSink;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy.build;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @ClassRule public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
+
+ @ClassRule
+ public static final MiniClusterWithClientResource miniClusterResource =
+ new MiniClusterWithClientResource(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
Review comment:
If you want to write an IT case you can use the new FileSink. I hope with 1.15 we can deprecate `StreamingFileSink`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r771369862
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.formats.csv.RowDataToCsvConverters.RowDataToCsvConverter.RowDataToCsvFormatConverterContext;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectWriter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import java.io.IOException;
+
+/** A simple {@link BulkWriter} implementation based on Jackson CSV transformations. */
+@PublicEvolving
+public class CsvBulkWriter<T> implements BulkWriter<T> {
+
+ private final FSDataOutputStream stream;
+ private final CsvMapper mapper;
+ private final Converter<T, JsonNode, RowDataToCsvFormatConverterContext> converter;
+ private final ObjectWriter csvWriter;
+ /** Reusable within the converter. * */
+ private final transient ObjectNode container;
Review comment:
You'd add `transient` only for `Serializable` classes. `BulkWriter` is not while `SerializationSchema` is. (BulkWriter is the better design)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 29e6025b084eb1c850ab3901c757dcfcb442c67e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289)
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
slinkydeveloper commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r768823822
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
I wonder if this one can be a projectable decoding format as well, given it roughly uses the same code for deserializing.
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.unixTimestampToLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class CsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
+ public void testProjectPushDown() throws Exception {
+ List<String> data = Arrays.asList("1,1,hi", "2,1,hello", "3,2,hello world");
+ createSourceTable("MyTable", data, "a bigint", "b int not null", "c varchar");
+ File sinkPath = createSinkTable("MySink", "a bigint", "c varchar");
+
+ String jsonPlan = tableEnv.getJsonPlan("insert into MySink select a, c from MyTable");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("1,hi", "2,hello", "3,hello world"), sinkPath);
+ }
+
+ @Test
+ public void testReadingMetadata() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.smallData3()),
+ new String[] {"a int", "b bigint", "m varchar metadata"},
+ new HashMap<String, String>() {
+ {
+ put("readable-metadata", "m:STRING");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a bigint", "m varchar");
+
+ String jsonPlan = tableEnv.getJsonPlan("insert into MySink select a, m from MyTable");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("1,Hi", "2,Hello", "3,Hello world"), sinkPath);
+ }
+
+ @Test
+ public void testFilterPushDown() throws Exception {
+ List<String> data = Arrays.asList("1,1,hi", "2,1,hello", "3,2,hello world");
+ createSourceTable("MyTable", data, "a bigint", "b int not null", "c varchar");
+ File sinkPath = createSinkTable("MySink", "a bigint", "b int", "c varchar");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select * from MyTable where a > 1");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("2,1,hello", "3,2,hello world"), sinkPath);
+ }
+
+ @Test
+ public void testPartitionPushDown() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.smallData3()),
+ new String[] {"a int", "p bigint", "c varchar"},
+ "p",
+ new HashMap<String, String>() {
+ {
+ put("partition-list", "p:1;p:2");
+ }
+ });
+ File sinkPath = createSinkTable("MySink", "a int", "p bigint", "c varchar");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select * from MyTable where p = 2");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("2,2,Hello", "3,2,Hello world"), sinkPath);
+ }
+
+ @Test
+ public void testWatermarkPushDown() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.data3WithTimestamp()),
+ new String[] {
+ "a int",
+ "b bigint",
+ "c varchar",
+ "ts timestamp(3)",
+ "watermark for ts as ts - interval '5' second"
+ },
+ new HashMap<String, String>() {
+ {
+ put("enable-watermark-push-down", "true");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a int", "b bigint", "ts timestamp(3)");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select a, b, ts from MyTable where b = 3");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(
+ Arrays.asList(
+ "4,3," + formatSqlTimestamp(4000L),
+ "5,3," + formatSqlTimestamp(5000L),
+ "6,3," + formatSqlTimestamp(6000L)),
+ sinkPath);
+ }
+
+ @Test
+ public void testPushDowns() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.data3WithTimestamp()),
+ new String[] {
+ "a int",
+ "b bigint",
+ "c varchar",
+ "ts timestamp(3)",
+ "watermark for ts as ts - interval '5' second"
+ },
+ "b",
+ new HashMap<String, String>() {
+ {
+ put("readable-metadata", "a:INT");
+ put("filterable-fields", "a");
+ put("enable-watermark-push-down", "true");
+ put("partition-list", "b:1;b:2;b:3;b:4;b:5;b:6");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a int", "ts timestamp(3)");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan(
+ "insert into MySink select a, ts from MyTable where b = 3 and a > 4");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(
+ Arrays.asList("5," + formatSqlTimestamp(5000L), "6," + formatSqlTimestamp(6000L)),
+ sinkPath);
+ }
+
+ private static String formatSqlTimestamp(long timestamp) {
+ return TimeFormats.SQL_TIMESTAMP_FORMAT.format(unixTimestampToLocalDateTime(timestamp));
+ }
Review comment:
Use `formatUnixTimestamp(long unixtime, TimeZone tz)` instead, as it also makes sure you use a fixed timezone and you don't get weird time zone issues when running on other machines :)
##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/utils/JsonPlanTestBase.java
##########
@@ -92,17 +92,27 @@ protected void createTestValuesSourceTable(
@Nullable String partitionFields,
Map<String, String> extraProperties) {
checkArgument(fieldNameAndTypes.length > 0);
- String partitionedBy =
- StringUtils.isNullOrWhitespaceOnly(partitionFields)
- ? ""
- : "\n partitioned by (" + partitionFields + ") \n";
+
String dataId = TestValuesTableFactory.registerData(data);
Map<String, String> properties = new HashMap<>();
properties.put("connector", "values");
properties.put("data-id", dataId);
properties.put("bounded", "true");
properties.put("disable-lookup", "true");
properties.putAll(extraProperties);
+ createTestSourceTable(tableName, fieldNameAndTypes, partitionFields, properties);
+ }
+
+ protected void createTestSourceTable(
+ String tableName,
+ String[] fieldNameAndTypes,
+ @Nullable String partitionFields,
+ Map<String, String> properties) {
+ checkArgument(fieldNameAndTypes.length > 0);
+ String partitionedBy =
+ StringUtils.isNullOrWhitespaceOnly(partitionFields)
+ ? ""
+ : "\n partitioned by (" + partitionFields + ") \n";
Review comment:
please remove this function usage and rewrite it using `Stream`s instead
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##########
@@ -125,128 +70,88 @@ public String factoryIdentifier() {
@Override
public Set<ConfigOption<?>> optionalOptions() {
- Set<ConfigOption<?>> options = new HashSet<>();
- options.add(FIELD_DELIMITER);
- options.add(DISABLE_QUOTE_CHARACTER);
- options.add(QUOTE_CHARACTER);
- options.add(ALLOW_COMMENTS);
- options.add(IGNORE_PARSE_ERRORS);
- options.add(ARRAY_ELEMENT_DELIMITER);
- options.add(ESCAPE_CHARACTER);
- options.add(NULL_LITERAL);
- return options;
- }
-
- // ------------------------------------------------------------------------
- // Validation
- // ------------------------------------------------------------------------
-
- static void validateFormatOptions(ReadableConfig tableOptions) {
- final boolean hasQuoteCharacter = tableOptions.getOptional(QUOTE_CHARACTER).isPresent();
- final boolean isDisabledQuoteCharacter = tableOptions.get(DISABLE_QUOTE_CHARACTER);
- if (isDisabledQuoteCharacter && hasQuoteCharacter) {
- throw new ValidationException(
- "Format cannot define a quote character and disabled quote character at the same time.");
- }
- // Validate the option value must be a single char.
- validateCharacterVal(tableOptions, FIELD_DELIMITER, true);
- validateCharacterVal(tableOptions, ARRAY_ELEMENT_DELIMITER);
- validateCharacterVal(tableOptions, QUOTE_CHARACTER);
- validateCharacterVal(tableOptions, ESCAPE_CHARACTER);
+ return CsvCommons.optionalOptions();
}
- /** Validates the option {@code option} value must be a Character. */
- private static void validateCharacterVal(
- ReadableConfig tableOptions, ConfigOption<String> option) {
- validateCharacterVal(tableOptions, option, false);
- }
+ @Override
+ public BulkDecodingFormat<RowData> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
- /**
- * Validates the option {@code option} value must be a Character.
- *
- * @param tableOptions the table options
- * @param option the config option
- * @param unescape whether to unescape the option value
- */
- private static void validateCharacterVal(
- ReadableConfig tableOptions, ConfigOption<String> option, boolean unescape) {
- if (tableOptions.getOptional(option).isPresent()) {
- final String value =
- unescape
- ? StringEscapeUtils.unescapeJava(tableOptions.get(option))
- : tableOptions.get(option);
- if (value.length() != 1) {
- throw new ValidationException(
- String.format(
- "Option '%s.%s' must be a string with single character, but was: %s",
- IDENTIFIER, option.key(), tableOptions.get(option)));
- }
- }
+ return new CsvBulkDecodingFormat(formatOptions);
}
- // ------------------------------------------------------------------------
- // Utilities
- // ------------------------------------------------------------------------
+ private static class CsvBulkDecodingFormat
+ implements BulkDecodingFormat<RowData>,
+ ProjectableDecodingFormat<BulkFormat<RowData, FileSourceSplit>> {
- private static void configureDeserializationSchema(
- ReadableConfig formatOptions, CsvRowDataDeserializationSchema.Builder schemaBuilder) {
- formatOptions
- .getOptional(FIELD_DELIMITER)
- .map(delimiter -> StringEscapeUtils.unescapeJava(delimiter).charAt(0))
- .ifPresent(schemaBuilder::setFieldDelimiter);
+ private final ReadableConfig formatOptions;
- if (formatOptions.get(DISABLE_QUOTE_CHARACTER)) {
- schemaBuilder.disableQuoteCharacter();
- } else {
- formatOptions
- .getOptional(QUOTE_CHARACTER)
- .map(quote -> quote.charAt(0))
- .ifPresent(schemaBuilder::setQuoteCharacter);
+ public CsvBulkDecodingFormat(ReadableConfig formatOptions) {
+ this.formatOptions = formatOptions;
}
- formatOptions.getOptional(ALLOW_COMMENTS).ifPresent(schemaBuilder::setAllowComments);
-
- formatOptions
- .getOptional(IGNORE_PARSE_ERRORS)
- .ifPresent(schemaBuilder::setIgnoreParseErrors);
-
- formatOptions
- .getOptional(ARRAY_ELEMENT_DELIMITER)
- .ifPresent(schemaBuilder::setArrayElementDelimiter);
-
- formatOptions
- .getOptional(ESCAPE_CHARACTER)
- .map(escape -> escape.charAt(0))
- .ifPresent(schemaBuilder::setEscapeCharacter);
+ @Override
+ // TODO: is it possible to avoid the cast with a reasonable effort?
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ public BulkFormat<RowData, FileSourceSplit> createRuntimeDecoder(
+ Context context, DataType physicalDataType, int[][] projections) {
+
+ final DataType projectedDataDype = Projection.of(projections).project(physicalDataType);
+ final RowType rowTypeProjected = (RowType) projectedDataDype.getLogicalType();
+
+ final RowType rowType = (RowType) physicalDataType.getLogicalType();
Review comment:
Name it `physicalRowType` so it makes the distinction with the above variable more clear
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770834164
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.unixTimestampToLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class CsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
+ public void testProjectPushDown() throws Exception {
Review comment:
Do you mean testing the new `CsvBulkWriter` and `CsvReaderFormat` not anything related to the ser/de schema-based (old) stuff? Another question - you point at the projection pushdown test, not sure if you meant it explicitly (new functionality?) or if it is a coincidence. Do we even have facilities for communicating pushdown from the streaming API?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770161770
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
I guess it could.
@AHeise, I believe we did not previously discuss this aspect. Do you think it is necessary to expand the scope of the projection pushdown support also to the Ser/De schema-based implementation for this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770169936
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.formats.csv.RowDataToCsvConverters.RowDataToCsvConverter.RowDataToCsvFormatConverterContext;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectWriter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import java.io.IOException;
+
+/** A simple {@link BulkWriter} implementation based on Jackson CSV transformations. */
+@PublicEvolving
+public class CsvBulkWriter<T> implements BulkWriter<T> {
+
+ private final FSDataOutputStream stream;
+ private final CsvMapper mapper;
+ private final Converter<T, JsonNode, RowDataToCsvFormatConverterContext> converter;
+ private final ObjectWriter csvWriter;
+ /** Reusable within the converter. * */
+ private final transient ObjectNode container;
Review comment:
This is in accordance with the design choice explicitly made in a similar situation in the recent addition:
https://github.com/apache/flink/commit/b1e436a109c6473b794f02ec0a853d1ae6df6c83#diff-58a105b8d92eadfb0af487ba6b2bce5a00955fceb2077e6b7f2d1fa01592a50bR84
One could argue that as it is a throw-away object that does not itself contain any configuration there is no strict necessity to ship it to the executors. Let me know what you think and if there are general guidelines for similar cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770603450
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
This would be orthogonal and can be done later. I'm also not sure if it's worth the hassle now: performance gain on a per-record projection is comparably small and we implicitly have it on all formats with reader schema.
However, I'd like you to verify if the current version actually works with projections, so can you please add an appropriate test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul merged pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul merged pull request #17598:
URL: https://github.com/apache/flink/pull/17598
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r792672907
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void> converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(), null) : null;
+ while (true) {
Review comment:
An arbitrary number of lines can be malformed. Let's say this is a single CSV file
```
a,b,c // header
1,2,3 // expected
1, // malformed
1,2,3,4 // malformed
5,6,7 // expected
```
Should produce [1,2,3] and [5,6,7] and ignore the rest. The number of malformed elements is unknown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r792645154
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void> converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(), null) : null;
+ while (true) {
+ try {
+ if (iterator.hasNext()) {
+ R nextElement = iterator.next();
+ return converter.convert(nextElement, null);
+ } else {
+ return null;
+ }
+ } catch (Throwable t) {
Review comment:
Is it necessary to catch all kinds of throwables here or is `IOException` enough? It can be dangerous to catch all kinds of exceptions i.e. if you handle a `InterruptedException` you also want to set the interrupted flag of the Thread.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void> converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(), null) : null;
+ while (true) {
+ try {
+ if (iterator.hasNext()) {
+ R nextElement = iterator.next();
+ return converter.convert(nextElement, null);
+ } else {
+ return null;
+ }
+ } catch (Throwable t) {
Review comment:
Is it necessary to catch all kinds of throwables here or is `IOException` enough? It can be dangerous to catch all kinds of exceptions i.e. if you handle an `InterruptedException` you also want to set the interrupted flag of the Thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r779611383
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/TableCsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.toLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class TableCsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
Review comment:
Yeah, `AbstractTestBase` which is JUnit4-based is used in 400 classes, I propose to address it in a separate effort :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r779598732
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
Review comment:
@fapaul I've added the explicit `Serializable` requirement to the `Converter` interface, because to me it looks like it is required. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r779599031
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormat.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
Review comment:
@fapaul maybe you have a good example?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r789562890
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @TempDir File outDir;
+
+ public static final MiniClusterExtension MINI_CLUSTER_RESOURCE =
+ new MiniClusterExtension(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ @RegisterExtension
+ public static AllCallbackWrapper<MiniClusterExtension> allCallbackWrapper =
+ new AllCallbackWrapper<>(MINI_CLUSTER_RESOURCE);
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
+ public void testBoundedTextFileSourceWithJackson() throws Exception {
+ writeFile(outDir, "data.csv", CSV_LINES);
+
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(outDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
+ // fromCollection is not bounded, using fromSequence instead
+ final List<CityPojo> pojosList = Arrays.asList(pojos); // needs to be Serializable
+ final DataStream<Integer> sequence =
+ env.fromSequence(0, pojos.length - 1).map(Long::intValue);
+ final DataStream<CityPojo> stream = sequence.map(pojosList::get).returns(CityPojo.class);
+
+ FileSink<CityPojo> sink =
+ FileSink.forBulkFormat(new Path(outDir.toURI()), factoryForPojo(CityPojo.class))
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+
+ stream.sinkTo(sink);
+ env.execute();
+
+ String[] result = getResultsFromSinkFiles(outDir);
+
+ assertThat(result).containsExactlyInAnyOrder(CSV_LINES);
Review comment:
Not sure what you mean, the call is checking that the lines are there in any order (containsExactlyIn**Any**Order)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
* f883156429a6510b06a05e58ddd77fb8e0554516 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* f883156429a6510b06a05e58ddd77fb8e0554516 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181)
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770164551
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.unixTimestampToLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class CsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
+ public void testProjectPushDown() throws Exception {
+ List<String> data = Arrays.asList("1,1,hi", "2,1,hello", "3,2,hello world");
+ createSourceTable("MyTable", data, "a bigint", "b int not null", "c varchar");
+ File sinkPath = createSinkTable("MySink", "a bigint", "c varchar");
+
+ String jsonPlan = tableEnv.getJsonPlan("insert into MySink select a, c from MyTable");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("1,hi", "2,hello", "3,hello world"), sinkPath);
+ }
+
+ @Test
+ public void testReadingMetadata() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.smallData3()),
+ new String[] {"a int", "b bigint", "m varchar metadata"},
+ new HashMap<String, String>() {
+ {
+ put("readable-metadata", "m:STRING");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a bigint", "m varchar");
+
+ String jsonPlan = tableEnv.getJsonPlan("insert into MySink select a, m from MyTable");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("1,Hi", "2,Hello", "3,Hello world"), sinkPath);
+ }
+
+ @Test
+ public void testFilterPushDown() throws Exception {
+ List<String> data = Arrays.asList("1,1,hi", "2,1,hello", "3,2,hello world");
+ createSourceTable("MyTable", data, "a bigint", "b int not null", "c varchar");
+ File sinkPath = createSinkTable("MySink", "a bigint", "b int", "c varchar");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select * from MyTable where a > 1");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("2,1,hello", "3,2,hello world"), sinkPath);
+ }
+
+ @Test
+ public void testPartitionPushDown() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.smallData3()),
+ new String[] {"a int", "p bigint", "c varchar"},
+ "p",
+ new HashMap<String, String>() {
+ {
+ put("partition-list", "p:1;p:2");
+ }
+ });
+ File sinkPath = createSinkTable("MySink", "a int", "p bigint", "c varchar");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select * from MyTable where p = 2");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("2,2,Hello", "3,2,Hello world"), sinkPath);
+ }
+
+ @Test
+ public void testWatermarkPushDown() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.data3WithTimestamp()),
+ new String[] {
+ "a int",
+ "b bigint",
+ "c varchar",
+ "ts timestamp(3)",
+ "watermark for ts as ts - interval '5' second"
+ },
+ new HashMap<String, String>() {
+ {
+ put("enable-watermark-push-down", "true");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a int", "b bigint", "ts timestamp(3)");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select a, b, ts from MyTable where b = 3");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(
+ Arrays.asList(
+ "4,3," + formatSqlTimestamp(4000L),
+ "5,3," + formatSqlTimestamp(5000L),
+ "6,3," + formatSqlTimestamp(6000L)),
+ sinkPath);
+ }
+
+ @Test
+ public void testPushDowns() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.data3WithTimestamp()),
+ new String[] {
+ "a int",
+ "b bigint",
+ "c varchar",
+ "ts timestamp(3)",
+ "watermark for ts as ts - interval '5' second"
+ },
+ "b",
+ new HashMap<String, String>() {
+ {
+ put("readable-metadata", "a:INT");
+ put("filterable-fields", "a");
+ put("enable-watermark-push-down", "true");
+ put("partition-list", "b:1;b:2;b:3;b:4;b:5;b:6");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a int", "ts timestamp(3)");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan(
+ "insert into MySink select a, ts from MyTable where b = 3 and a > 4");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(
+ Arrays.asList("5," + formatSqlTimestamp(5000L), "6," + formatSqlTimestamp(6000L)),
+ sinkPath);
+ }
+
+ private static String formatSqlTimestamp(long timestamp) {
+ return TimeFormats.SQL_TIMESTAMP_FORMAT.format(unixTimestampToLocalDateTime(timestamp));
+ }
Review comment:
Not sure what you mean. I can see the `formatUnixTimestamp` only in JavaScript libraries in Flink's code base. Or is your point that the timezone should necessarily be also taken into account and `formatUnixTimestamp(long unixtime, TimeZone tz)` is what you propose to call my method instead of `formatSqlTimestamp`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770834164
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.unixTimestampToLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class CsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
+ public void testProjectPushDown() throws Exception {
Review comment:
Will work on that, but first a small clarification:
Do you mean testing the new `CsvBulkWriter` and `CsvReaderFormat` not anything related to the ser/de schema-based (old) stuff? Another question - you point at the projection pushdown test, not sure if you meant it explicitly (new functionality?) or if it is a coincidence. Do we even have facilities for communicating pushdown from the streaming API?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770835661
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormat.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
Review comment:
Do you have a good inspirational example for this kind of instruction in our codebase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 29e6025b084eb1c850ab3901c757dcfcb442c67e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289)
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r771443163
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
Review comment:
@AHeise I decoupled the writer from `RowData` in a separate hotfix [`8074cb4` (#17598)](https://github.com/apache/flink/pull/17598/commits/8074cb4ace3b77d71d67926d289d7322ff3eb4c2)
It makes instantiation in the factory a bit awkward, but it works. What do you think?
If we keep it, how about putting a Serializable requirement also explicitly on the context type of the `Converter` interface?
https://github.com/apache/flink/pull/17598/files#diff-eeda89bce5e097c1019fe734e379cc5fb7a2c2f5e528d5b26bc715118eece38eR30
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770696351
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##########
@@ -19,103 +19,48 @@
package org.apache.flink.formats.csv;
import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.common.serialization.SerializationSchema;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.configuration.ReadableConfig;
-import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.impl.StreamFormatAdapter;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.formats.common.Converter;
import org.apache.flink.table.connector.ChangelogMode;
-import org.apache.flink.table.connector.format.DecodingFormat;
-import org.apache.flink.table.connector.format.EncodingFormat;
-import org.apache.flink.table.connector.sink.DynamicTableSink;
-import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.Projection;
+import org.apache.flink.table.connector.format.BulkDecodingFormat;
+import org.apache.flink.table.connector.format.ProjectableDecodingFormat;
+import org.apache.flink.table.connector.source.DynamicTableSource.Context;
import org.apache.flink.table.data.RowData;
-import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.BulkReaderFormatFactory;
import org.apache.flink.table.factories.DynamicTableFactory;
-import org.apache.flink.table.factories.FactoryUtil;
-import org.apache.flink.table.factories.SerializationFormatFactory;
import org.apache.flink.table.types.DataType;
import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
import org.apache.commons.lang3.StringEscapeUtils;
import java.util.Collections;
-import java.util.HashSet;
import java.util.Set;
import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
-import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
-/**
- * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
- * and {@link DeserializationSchema}.
- */
+/** CSV format factory for file system. */
+//TODO: Rename back to CsvFileSystemFormatFactory? @Arvid
Review comment:
I guess we could pass the `OutputStream` instead of the `FSDataOutputStream`, but that would mean adding a check via `isinstanceof` in the `finish()` method to determine if the `sync()` call is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02355cdca67d73500f5251834bc07df18ec1f2a8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r779500430
##########
File path: flink-formats/flink-format-common/src/main/java/org/apache/flink/formats/common/Converter.java
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.common;
+
+import java.io.Serializable;
+
+/**
+ * A generic interface for converting data types.
+ *
+ * @param <From> The type of the element to be converted.
+ * @param <To> The output type.
+ * @param <C> The context for passing optional conversion instructions.
+ */
+public interface Converter<From, To, C> extends Serializable {
Review comment:
Good point, is there a preference which one to use in this particular case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 35eb6edee6ce7d07bca66782ed264bdb00fbdc57 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 35eb6edee6ce7d07bca66782ed264bdb00fbdc57 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052)
* dc499251534da1f28801ef639c8eb8ee261de3c8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e5445170476fae8081037143c62c36318677ad6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6aea9ddf28bf11ed35c87ef57e477fecca94f266 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192)
* 4e5445170476fae8081037143c62c36318677ad6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-tbd][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9aa8115b9b3fa8cc853f7821c29f939870c79e5d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r789565174
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @TempDir File outDir;
+
+ public static final MiniClusterExtension MINI_CLUSTER_RESOURCE =
+ new MiniClusterExtension(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ @RegisterExtension
+ public static AllCallbackWrapper<MiniClusterExtension> allCallbackWrapper =
+ new AllCallbackWrapper<>(MINI_CLUSTER_RESOURCE);
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
+ public void testBoundedTextFileSourceWithJackson() throws Exception {
+ writeFile(outDir, "data.csv", CSV_LINES);
+
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(outDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
+ // fromCollection is not bounded, using fromSequence instead
+ final List<CityPojo> pojosList = Arrays.asList(pojos); // needs to be Serializable
+ final DataStream<Integer> sequence =
+ env.fromSequence(0, pojos.length - 1).map(Long::intValue);
+ final DataStream<CityPojo> stream = sequence.map(pojosList::get).returns(CityPojo.class);
+
+ FileSink<CityPojo> sink =
+ FileSink.forBulkFormat(new Path(outDir.toURI()), factoryForPojo(CityPojo.class))
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+
+ stream.sinkTo(sink);
+ env.execute();
+
+ String[] result = getResultsFromSinkFiles(outDir);
+
+ assertThat(result).containsExactlyInAnyOrder(CSV_LINES);
+ }
+
+ @NotNull
+ private String[] getResultsFromSinkFiles(File outDir) throws IOException {
+ final Map<File, String> contents = getFileContentByPath(outDir);
+
+ List<String> resultList =
+ contents.entrySet().stream()
+ .flatMap(e -> Arrays.stream(e.getValue().split("\n")))
+ .collect(Collectors.toList());
+
+ String[] result = resultList.toArray(new String[0]);
+ return result;
+ }
+
+ private static <T> BulkWriter.Factory<T> factoryForPojo(Class<T> pojoClass) {
+ final Converter<T, T, Void> converter = (value, context) -> value;
+ final CsvMapper csvMapper = new CsvMapper();
+ final CsvSchema schema = csvMapper.schemaFor(pojoClass).withoutQuoteChar();
+ return (out) -> new CsvBulkWriter<>(csvMapper, schema, converter, null, out);
+ }
+
+ private static Map<File, String> getFileContentByPath(File directory) throws IOException {
+ Map<File, String> contents = new HashMap<>(4);
+
+ final Collection<File> filesInBucket = FileUtils.listFiles(directory, null, true);
+ for (File file : filesInBucket) {
+ contents.put(file, FileUtils.readFileToString(file));
+ }
+ return contents;
+ }
+
+ @JsonPropertyOrder({
+ "city",
+ "lat",
+ "lng",
+ "country",
+ "iso2",
+ "adminName",
+ "capital",
+ "population"
+ })
+ public static class CityPojo implements Serializable {
+ public String city;
+ public BigDecimal lat;
+ public BigDecimal lng;
+ public String country;
+ public String iso2;
+ public String adminName;
+ public String capital;
+ public long population;
+
+ public CityPojo() {};
+
+ public CityPojo(
+ String city,
+ BigDecimal lat,
+ BigDecimal lng,
+ String country,
+ String iso2,
+ String admin_name,
+ String capital,
+ long population) {
+ this.city = city;
+ this.lat = lat;
+ this.lng = lng;
+ this.country = country;
+ this.iso2 = iso2;
+ this.adminName = admin_name;
+ this.capital = capital;
+ this.population = population;
+ }
+
+ @Override
+ public String toString() {
+ return "CitiesPojo{"
+ + "city='"
+ + city
+ + '\''
+ + ", lat="
+ + lat
+ + ", lng="
+ + lng
+ + ", country='"
+ + country
+ + '\''
+ + ", iso2='"
+ + iso2
+ + '\''
+ + ", adminName='"
+ + adminName
+ + '\''
+ + ", capital='"
+ + capital
+ + '\''
+ + ", population="
+ + population
+ + '}';
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ CityPojo that = (CityPojo) o;
+ return population == that.population
+ && Objects.equals(city, that.city)
+ && Objects.equals(lat, that.lat)
+ && Objects.equals(lng, that.lng)
+ && Objects.equals(country, that.country)
+ && Objects.equals(iso2, that.iso2)
+ && Objects.equals(adminName, that.adminName)
+ && Objects.equals(capital, that.capital);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(city, lat, lng, country, iso2, adminName, capital, population);
+ }
+ }
+
+ private static <T> List<T> initializeSourceAndReadData(File testDir, StreamFormat<T> csvFormat)
+ throws Exception {
+ final FileSource<T> source =
+ FileSource.forRecordStreamFormat(csvFormat, Path.fromLocalFile(testDir)).build();
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
+ final DataStream<T> stream =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "file-source");
+
+ return getResultsFromStream(stream);
+ }
+
+ @NotNull
+ private static <T> List<T> getResultsFromStream(DataStream<T> stream) throws Exception {
+ final ClientAndIterator<T> client =
+ DataStreamUtils.collectWithClient(stream, "Bounded Results Fetch");
+
+ final List<T> result = new ArrayList<>();
+ while (client.iterator.hasNext()) {
+ T next = client.iterator.next();
+ result.add(next);
+ }
+ return result;
+ }
+
+ // ------------------------------------------------------------------------
+ // test data
+ // ------------------------------------------------------------------------
+
+ private static final String[] CSV_LINES =
Review comment:
Moved static. I don't quite understand what you have in mind with regarding the method.
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.AllCallbackWrapper;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterExtension;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @TempDir File outDir;
+
+ public static final MiniClusterExtension MINI_CLUSTER_RESOURCE =
+ new MiniClusterExtension(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
+
+ @RegisterExtension
+ public static AllCallbackWrapper<MiniClusterExtension> allCallbackWrapper =
+ new AllCallbackWrapper<>(MINI_CLUSTER_RESOURCE);
+
+ // ------------------------------------------------------------------------
+ // test cases
+ // ------------------------------------------------------------------------
+ @Test
+ public void testBoundedTextFileSourceWithJackson() throws Exception {
+ writeFile(outDir, "data.csv", CSV_LINES);
+
+ final CsvReaderFormat<CityPojo> csvFormat = CsvReaderFormat.forPojo(CityPojo.class);
+ final List<CityPojo> result = initializeSourceAndReadData(outDir, csvFormat);
+
+ assertEquals(Arrays.asList(pojos), result);
+ }
+
+ @Test
+ public void testCustomBulkWriter() throws Exception {
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
+ // fromCollection is not bounded, using fromSequence instead
+ final List<CityPojo> pojosList = Arrays.asList(pojos); // needs to be Serializable
+ final DataStream<Integer> sequence =
+ env.fromSequence(0, pojos.length - 1).map(Long::intValue);
+ final DataStream<CityPojo> stream = sequence.map(pojosList::get).returns(CityPojo.class);
+
+ FileSink<CityPojo> sink =
+ FileSink.forBulkFormat(new Path(outDir.toURI()), factoryForPojo(CityPojo.class))
+ .withBucketAssigner(new BasePathBucketAssigner<>())
+ .build();
+
+ stream.sinkTo(sink);
+ env.execute();
+
+ String[] result = getResultsFromSinkFiles(outDir);
+
+ assertThat(result).containsExactlyInAnyOrder(CSV_LINES);
+ }
+
+ @NotNull
+ private String[] getResultsFromSinkFiles(File outDir) throws IOException {
+ final Map<File, String> contents = getFileContentByPath(outDir);
+
+ List<String> resultList =
+ contents.entrySet().stream()
+ .flatMap(e -> Arrays.stream(e.getValue().split("\n")))
+ .collect(Collectors.toList());
+
+ String[] result = resultList.toArray(new String[0]);
+ return result;
+ }
+
+ private static <T> BulkWriter.Factory<T> factoryForPojo(Class<T> pojoClass) {
+ final Converter<T, T, Void> converter = (value, context) -> value;
+ final CsvMapper csvMapper = new CsvMapper();
+ final CsvSchema schema = csvMapper.schemaFor(pojoClass).withoutQuoteChar();
+ return (out) -> new CsvBulkWriter<>(csvMapper, schema, converter, null, out);
+ }
+
+ private static Map<File, String> getFileContentByPath(File directory) throws IOException {
+ Map<File, String> contents = new HashMap<>(4);
+
+ final Collection<File> filesInBucket = FileUtils.listFiles(directory, null, true);
+ for (File file : filesInBucket) {
+ contents.put(file, FileUtils.readFileToString(file));
+ }
+ return contents;
+ }
+
+ @JsonPropertyOrder({
+ "city",
+ "lat",
+ "lng",
+ "country",
+ "iso2",
+ "adminName",
+ "capital",
+ "population"
+ })
+ public static class CityPojo implements Serializable {
+ public String city;
+ public BigDecimal lat;
+ public BigDecimal lng;
+ public String country;
+ public String iso2;
+ public String adminName;
+ public String capital;
+ public long population;
+
+ public CityPojo() {};
+
+ public CityPojo(
+ String city,
+ BigDecimal lat,
+ BigDecimal lng,
+ String country,
+ String iso2,
+ String admin_name,
+ String capital,
+ long population) {
+ this.city = city;
+ this.lat = lat;
+ this.lng = lng;
+ this.country = country;
+ this.iso2 = iso2;
+ this.adminName = admin_name;
+ this.capital = capital;
+ this.population = population;
+ }
+
+ @Override
+ public String toString() {
+ return "CitiesPojo{"
+ + "city='"
+ + city
+ + '\''
+ + ", lat="
+ + lat
+ + ", lng="
+ + lng
+ + ", country='"
+ + country
+ + '\''
+ + ", iso2='"
+ + iso2
+ + '\''
+ + ", adminName='"
+ + adminName
+ + '\''
+ + ", capital='"
+ + capital
+ + '\''
+ + ", population="
+ + population
+ + '}';
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ CityPojo that = (CityPojo) o;
+ return population == that.population
+ && Objects.equals(city, that.city)
+ && Objects.equals(lat, that.lat)
+ && Objects.equals(lng, that.lng)
+ && Objects.equals(country, that.country)
+ && Objects.equals(iso2, that.iso2)
+ && Objects.equals(adminName, that.adminName)
+ && Objects.equals(capital, that.capital);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(city, lat, lng, country, iso2, adminName, capital, population);
+ }
+ }
+
+ private static <T> List<T> initializeSourceAndReadData(File testDir, StreamFormat<T> csvFormat)
+ throws Exception {
+ final FileSource<T> source =
+ FileSource.forRecordStreamFormat(csvFormat, Path.fromLocalFile(testDir)).build();
+
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(PARALLELISM);
+ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
+
+ final DataStream<T> stream =
+ env.fromSource(source, WatermarkStrategy.noWatermarks(), "file-source");
+
+ return getResultsFromStream(stream);
+ }
+
+ @NotNull
+ private static <T> List<T> getResultsFromStream(DataStream<T> stream) throws Exception {
+ final ClientAndIterator<T> client =
+ DataStreamUtils.collectWithClient(stream, "Bounded Results Fetch");
+
+ final List<T> result = new ArrayList<>();
+ while (client.iterator.hasNext()) {
+ T next = client.iterator.next();
+ result.add(next);
+ }
+ return result;
+ }
+
+ // ------------------------------------------------------------------------
+ // test data
+ // ------------------------------------------------------------------------
+
+ private static final String[] CSV_LINES =
Review comment:
Moved static. I don't quite understand what you have in mind regarding the method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
* f883156429a6510b06a05e58ddd77fb8e0554516 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181)
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r792676956
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void> converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(), null) : null;
+ while (true) {
Review comment:
I see then I'd propose to rewrite the loop to
```java
while (iterator.hasNext()) {
try {
R nextElement = iterator.next();
return converter.convert(nextElement,
} catch (....) {
}
}
return null;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
}, {
"hash" : "f51dee82fb528a7ae6f883c46be76b3e1b3af6c4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f51dee82fb528a7ae6f883c46be76b3e1b3af6c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183)
* f51dee82fb528a7ae6f883c46be76b3e1b3af6c4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r792705881
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFileFormatFactory.java
##########
@@ -109,17 +110,21 @@ public CsvBulkDecodingFormat(ReadableConfig formatOptions) {
final RowType physicalRowType = (RowType) physicalDataType.getLogicalType();
final CsvSchema schema = buildCsvSchema(physicalRowType, formatOptions);
+ final boolean ignoreParseErrors =
+ formatOptions.getOptional(IGNORE_PARSE_ERRORS).isPresent();
final Converter<JsonNode, RowData, Void> converter =
(Converter)
- new CsvToRowDataConverters(false)
+ new CsvToRowDataConverters(ignoreParseErrors)
.createRowConverter(projectedRowType, true);
- return new StreamFormatAdapter<>(
+ CsvReaderFormat<RowData> csvReaderFormat =
new CsvReaderFormat<>(
new CsvMapper(),
schema,
JsonNode.class,
converter,
- context.createTypeInformation(projectedDataType)));
+ context.createTypeInformation(projectedDataType));
+ csvReaderFormat.setIgnoreParseErrors(ignoreParseErrors);
Review comment:
Good point. I think the best way here is to use the `wither` pattern [1]. You can add a method like
`public CsvReaderFormat<T> withIgnoreParseErrors()`
that creates a new `CsvReaderFormat` with the flag that it ignores the errors. By solely using `fromSchema` or `fromPojo` nothing changes. Users can now do `CsvReaderFormat.fromPojo(Pojo.class).withIgnoreParseErrors()`
[1] https://projectlombok.org/features/With
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47b6d1194869dc00a6d326bf87127da8c6d667a3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 29e6025b084eb1c850ab3901c757dcfcb442c67e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289)
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
* 949b964bcb65db05a9c903a28f59f933300a7935 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
slinkydeveloper commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r773813519
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
Ok but then open a followup issue for that, because this inconsistency between the `CsvFormatFactory` and `CsvSerDeSchemaFactory` is confusing and doesn't really make sense to exist, as either both support projections, or none of them does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 949b964bcb65db05a9c903a28f59f933300a7935 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337)
* 02355cdca67d73500f5251834bc07df18ec1f2a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r779513508
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectWriter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import java.io.IOException;
+
+// TODO: ? PublicEvolving?
+/** A simple {@link BulkWriter} implementation based on Jackson CSV transformations. */
+class CsvBulkWriter<T, R, C> implements BulkWriter<T> {
+
+ private final FSDataOutputStream stream;
+ private final Converter<T, R, C> converter;
+ private final C converterContext;
+ private final ObjectWriter csvWriter;
+
+ CsvBulkWriter(
+ CsvMapper mapper,
+ CsvSchema schema,
+ Converter<T, R, C> converter,
+ C converterContext,
+ FSDataOutputStream stream) {
+ this.converter = converter;
+ this.converterContext = converterContext;
+ this.stream = stream;
Review comment:
As far as the documentation states, it is supposed to be handled by the new sink machinery
https://github.com/apache/flink/blob/c6997c97c575d334679915c328792b8a3067cfb5/flink-core/src/main/java/org/apache/flink/api/common/serialization/BulkWriter.java#L69
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r780379983
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/TableCsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.toLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class TableCsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
Review comment:
Reverted back to the `JsonPlanTestBase`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 744b699a82b4640652050eb75eecaf78b2f10c2c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105)
* e089768e55137235bd5b85a0b6b97b00dd0d23f9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
* f883156429a6510b06a05e58ddd77fb8e0554516 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7e097463582d225c33b56c3d2f067a3954867a11 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770126659
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormat.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
+ * @param <T>
+ */
+@PublicEvolving
+public class CsvFormat<T> extends SimpleStreamFormat<T> {
+
+ private static final long serialVersionUID = 1L;
+
+ private final CsvMapper mapper;
+ private final CsvSchema schema;
+ private final Class<Object> rootType;
+ private final Converter<Object, T, Void> converter;
+ private final TypeInformation<T> typeInformation;
+
+ @SuppressWarnings("unchecked")
+ <R> CsvFormat(
+ CsvMapper mapper,
+ CsvSchema schema,
+ Class<R> rootType,
+ Converter<R, T, Void> converter,
+ TypeInformation<T> typeInformation) {
+ this.mapper = checkNotNull(mapper);
+ this.schema = schema;
+ this.rootType = (Class<Object>) checkNotNull(rootType);
+ this.typeInformation = checkNotNull(typeInformation);
+ this.converter = (Converter<Object, T, Void>) checkNotNull(converter);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} using a {@code CsvSchema}.
+ *
+ * @param schema The Jackson CSV schema configured for parsing specific CSV files.
+ * @param typeInformation The Flink type descriptor of the returned elements.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(CsvSchema schema, TypeInformation<T> typeInformation) {
+ return from(new CsvMapper(), schema, typeInformation);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} using a {@code CsvSchema} and a pre-created {@code CsvMapper}.
+ *
+ * @param mapper The pre-created {@code CsvMapper}.
+ * @param schema The Jackson CSV schema configured for parsing specific CSV files.
+ * @param typeInformation The Flink type descriptor of the returned elements.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(
+ CsvMapper mapper, CsvSchema schema, TypeInformation<T> typeInformation) {
+ return new CsvFormat<>(
+ mapper,
+ schema,
+ typeInformation.getTypeClass(),
+ (value, context) -> value,
+ typeInformation);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} for reading CSV files mapped to the provided POJO class
+ * definition.
+ *
+ * @param pojoType The type class of the POJO.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(Class<T> pojoType) {
+ CsvMapper mapper = new CsvMapper();
+ return from(mapper, mapper.schemaFor(pojoType), TypeInformation.of(pojoType));
+ }
+
+ @Override
+ public StreamFormat.Reader<T> createReader(Configuration config, FSDataInputStream stream)
+ throws IOException {
+ return new Reader<>(mapper.readerFor(rootType).with(schema).readValues(stream), converter);
+ }
+
+ @Override
+ public TypeInformation<T> getProducedType() {
+ return typeInformation;
+ }
+
+ // ------------------------------------------------------------------------
+
+ /** The actual reader for the {@code CsvFormat}. */
+ public static final class Reader<R, T> implements StreamFormat.Reader<T> {
Review comment:
Good point. Copy-paste inconsistency from the `TextLineFormat` format "template".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770475079
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##########
@@ -125,128 +70,88 @@ public String factoryIdentifier() {
@Override
public Set<ConfigOption<?>> optionalOptions() {
- Set<ConfigOption<?>> options = new HashSet<>();
- options.add(FIELD_DELIMITER);
- options.add(DISABLE_QUOTE_CHARACTER);
- options.add(QUOTE_CHARACTER);
- options.add(ALLOW_COMMENTS);
- options.add(IGNORE_PARSE_ERRORS);
- options.add(ARRAY_ELEMENT_DELIMITER);
- options.add(ESCAPE_CHARACTER);
- options.add(NULL_LITERAL);
- return options;
- }
-
- // ------------------------------------------------------------------------
- // Validation
- // ------------------------------------------------------------------------
-
- static void validateFormatOptions(ReadableConfig tableOptions) {
- final boolean hasQuoteCharacter = tableOptions.getOptional(QUOTE_CHARACTER).isPresent();
- final boolean isDisabledQuoteCharacter = tableOptions.get(DISABLE_QUOTE_CHARACTER);
- if (isDisabledQuoteCharacter && hasQuoteCharacter) {
- throw new ValidationException(
- "Format cannot define a quote character and disabled quote character at the same time.");
- }
- // Validate the option value must be a single char.
- validateCharacterVal(tableOptions, FIELD_DELIMITER, true);
- validateCharacterVal(tableOptions, ARRAY_ELEMENT_DELIMITER);
- validateCharacterVal(tableOptions, QUOTE_CHARACTER);
- validateCharacterVal(tableOptions, ESCAPE_CHARACTER);
+ return CsvCommons.optionalOptions();
}
- /** Validates the option {@code option} value must be a Character. */
- private static void validateCharacterVal(
- ReadableConfig tableOptions, ConfigOption<String> option) {
- validateCharacterVal(tableOptions, option, false);
- }
+ @Override
+ public BulkDecodingFormat<RowData> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
- /**
- * Validates the option {@code option} value must be a Character.
- *
- * @param tableOptions the table options
- * @param option the config option
- * @param unescape whether to unescape the option value
- */
- private static void validateCharacterVal(
- ReadableConfig tableOptions, ConfigOption<String> option, boolean unescape) {
- if (tableOptions.getOptional(option).isPresent()) {
- final String value =
- unescape
- ? StringEscapeUtils.unescapeJava(tableOptions.get(option))
- : tableOptions.get(option);
- if (value.length() != 1) {
- throw new ValidationException(
- String.format(
- "Option '%s.%s' must be a string with single character, but was: %s",
- IDENTIFIER, option.key(), tableOptions.get(option)));
- }
- }
+ return new CsvBulkDecodingFormat(formatOptions);
}
- // ------------------------------------------------------------------------
- // Utilities
- // ------------------------------------------------------------------------
+ private static class CsvBulkDecodingFormat
+ implements BulkDecodingFormat<RowData>,
+ ProjectableDecodingFormat<BulkFormat<RowData, FileSourceSplit>> {
- private static void configureDeserializationSchema(
- ReadableConfig formatOptions, CsvRowDataDeserializationSchema.Builder schemaBuilder) {
- formatOptions
- .getOptional(FIELD_DELIMITER)
- .map(delimiter -> StringEscapeUtils.unescapeJava(delimiter).charAt(0))
- .ifPresent(schemaBuilder::setFieldDelimiter);
+ private final ReadableConfig formatOptions;
- if (formatOptions.get(DISABLE_QUOTE_CHARACTER)) {
- schemaBuilder.disableQuoteCharacter();
- } else {
- formatOptions
- .getOptional(QUOTE_CHARACTER)
- .map(quote -> quote.charAt(0))
- .ifPresent(schemaBuilder::setQuoteCharacter);
+ public CsvBulkDecodingFormat(ReadableConfig formatOptions) {
+ this.formatOptions = formatOptions;
}
- formatOptions.getOptional(ALLOW_COMMENTS).ifPresent(schemaBuilder::setAllowComments);
-
- formatOptions
- .getOptional(IGNORE_PARSE_ERRORS)
- .ifPresent(schemaBuilder::setIgnoreParseErrors);
-
- formatOptions
- .getOptional(ARRAY_ELEMENT_DELIMITER)
- .ifPresent(schemaBuilder::setArrayElementDelimiter);
-
- formatOptions
- .getOptional(ESCAPE_CHARACTER)
- .map(escape -> escape.charAt(0))
- .ifPresent(schemaBuilder::setEscapeCharacter);
+ @Override
+ // TODO: is it possible to avoid the cast with a reasonable effort?
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ public BulkFormat<RowData, FileSourceSplit> createRuntimeDecoder(
+ Context context, DataType physicalDataType, int[][] projections) {
+
+ final DataType projectedDataDype = Projection.of(projections).project(physicalDataType);
+ final RowType rowTypeProjected = (RowType) projectedDataDype.getLogicalType();
+
+ final RowType rowType = (RowType) physicalDataType.getLogicalType();
Review comment:
Thanks, makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-996095786
@AHeise @slinkydeveloper thank you for the feedback.
I addressed your comments where there were no clarifications needed. I am still working on a couple more tests for the DataStream API, but apart from that this PR is ready for another pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155)
* 29e6025b084eb1c850ab3901c757dcfcb442c67e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r771443163
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
Review comment:
@AHeise I decoupled the writer from `RowData` in a separate hotfix [`8074cb4` (#17598)](https://github.com/apache/flink/pull/17598/commits/8074cb4ace3b77d71d67926d289d7322ff3eb4c2)
It makes instantiation in the factory a bit awkward, but it works. What do you think?
If we keep it, how about putting a Serializable requirement also explicitly on the context type of the Converter interface?
https://github.com/apache/flink/pull/17598/files#diff-eeda89bce5e097c1019fe734e379cc5fb7a2c2f5e528d5b26bc715118eece38eR30
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r779588908
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/TableCsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.toLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class TableCsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
Review comment:
This test is largely based on utilities from the `JsonPlanTestBase` and `AbstractTestBase` base classes. I would prefer to leave the migration of those utility classes for table API to someone who works on Table/SQL. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 744b699a82b4640652050eb75eecaf78b2f10c2c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105)
* e089768e55137235bd5b85a0b6b97b00dd0d23f9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r779815815
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/TableCsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.toLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class TableCsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
Review comment:
`JsonPlanTestBase` is also used in 28 tests :sweat_smile: . I created a separate `TableFormatTestBase` based on JUnit5 and migrated my test to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 744b699a82b4640652050eb75eecaf78b2f10c2c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105)
* e089768e55137235bd5b85a0b6b97b00dd0d23f9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02355cdca67d73500f5251834bc07df18ec1f2a8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981)
* 35eb6edee6ce7d07bca66782ed264bdb00fbdc57 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 35eb6edee6ce7d07bca66782ed264bdb00fbdc57 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052)
* dc499251534da1f28801ef639c8eb8ee261de3c8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc499251534da1f28801ef639c8eb8ee261de3c8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064)
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 744b699a82b4640652050eb75eecaf78b2f10c2c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r779584052
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/DataStreamCsvITCase.java
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink;
+import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import org.apache.flink.streaming.api.operators.StreamSink;
+import org.apache.flink.streaming.api.operators.collect.ClientAndIterator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.FunctionWithException;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.UUID;
+
+import static org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy.build;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** MiniCluster-based integration tests CSV data format. */
+public class DataStreamCsvITCase extends TestLogger {
+
+ private static final int PARALLELISM = 4;
+
+ @ClassRule public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
+
+ @ClassRule
+ public static final MiniClusterWithClientResource miniClusterResource =
+ new MiniClusterWithClientResource(
+ new MiniClusterResourceConfiguration.Builder()
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(PARALLELISM)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .withHaLeadershipControl()
+ .build());
Review comment:
Good point, I did not realize that `StreamExecutionEnvironment.getExecutionEnvironment();` does not need it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r779616445
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/TableCsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.toLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class TableCsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
Review comment:
I mean you do not have to touch `AbstractTestBase` it does not offer much besides a MiniCluster which you can also get with the existing extension. The only effort would be to rework `JsonPlanTestBase`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-1007555875
@fapaul I addressed all of the comments. The only remaining open points from my perspective:
- clarify if imposing a strict `Serializable` requirement on the `Converter` interface is desirable. I believe Arvid mentioned that shifting this concern to factories was described as "better design" (we do not have factories for the existing converters though).
- clarify if there should also be an explicit `CsvWriterFactory` implementation of `BulkWriter.Factory<T>` for consistency (similar to `ParquetWriterFactory<T>` and `AvroWriterFactory<T>`)
- open a follow-up ticket for projection pushdown support for the previously existing ser/de-schema-based implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17598: [WIP][FLINK-tbd][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282833
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 9aa8115b9b3fa8cc853f7821c29f939870c79e5d (Thu Oct 28 22:53:04 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **Invalid pull request title: No valid Jira ID provided**
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r792685137
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void> converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(), null) : null;
+ while (true) {
+ try {
+ if (iterator.hasNext()) {
+ R nextElement = iterator.next();
+ return converter.convert(nextElement, null);
+ } else {
+ return null;
+ }
+ } catch (Throwable t) {
Review comment:
Ok let's keep it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
}, {
"hash" : "f51dee82fb528a7ae6f883c46be76b3e1b3af6c4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30255",
"triggerID" : "f51dee82fb528a7ae6f883c46be76b3e1b3af6c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183)
* f51dee82fb528a7ae6f883c46be76b3e1b3af6c4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30255)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r792689731
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void> converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(), null) : null;
+ while (true) {
+ try {
+ if (iterator.hasNext()) {
+ R nextElement = iterator.next();
+ return converter.convert(nextElement, null);
+ } else {
+ return null;
+ }
+ } catch (Throwable t) {
Review comment:
But a good point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r773800347
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
@slinkydeveloper
I am not sure about this actually. The focus of this PR is
**Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
I would rather not touch the existing Ser/De schema-based functionality in this scope. Nothing will break if we won't support pushdown in the non-filesystem connectors for now and to me, it sounds like an improvement that should be done in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0e3bc85342e7ae33117597931baa83f7c78c19a4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687)
* 47b6d1194869dc00a6d326bf87127da8c6d667a3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
}, {
"hash" : "f51dee82fb528a7ae6f883c46be76b3e1b3af6c4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30255",
"triggerID" : "f51dee82fb528a7ae6f883c46be76b3e1b3af6c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* f51dee82fb528a7ae6f883c46be76b3e1b3af6c4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30255)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r792689731
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void> converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(), null) : null;
+ while (true) {
+ try {
+ if (iterator.hasNext()) {
+ R nextElement = iterator.next();
+ return converter.convert(nextElement, null);
+ } else {
+ return null;
+ }
+ } catch (Throwable t) {
Review comment:
But good point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
}, {
"hash" : "f51dee82fb528a7ae6f883c46be76b3e1b3af6c4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f51dee82fb528a7ae6f883c46be76b3e1b3af6c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183)
* f51dee82fb528a7ae6f883c46be76b3e1b3af6c4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919)
* f883156429a6510b06a05e58ddd77fb8e0554516 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29919",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
}, {
"hash" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30181",
"triggerID" : "f883156429a6510b06a05e58ddd77fb8e0554516",
"triggerType" : "PUSH"
}, {
"hash" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183",
"triggerID" : "e7827ee1a27bf459ce67e861dc9d0bf90691c2fa",
"triggerType" : "PUSH"
}, {
"hash" : "f51dee82fb528a7ae6f883c46be76b3e1b3af6c4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f51dee82fb528a7ae6f883c46be76b3e1b3af6c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* e7827ee1a27bf459ce67e861dc9d0bf90691c2fa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30183)
* f51dee82fb528a7ae6f883c46be76b3e1b3af6c4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r789576905
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/TableCsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.toLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class TableCsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
+ public void testProjectPushDown() throws Exception {
Review comment:
I've added support for the new implementation of the formats:
https://github.com/afedulov/flink/blob/744b699a82b4640652050eb75eecaf78b2f10c2c/flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFileFormatFactory.java#L104
The discussion was about also backporting/adding this feature to the already-existing SerDe-schema-based implementation within the scope of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28337",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
}, {
"hash" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28981",
"triggerID" : "02355cdca67d73500f5251834bc07df18ec1f2a8",
"triggerType" : "PUSH"
}, {
"hash" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29052",
"triggerID" : "35eb6edee6ce7d07bca66782ed264bdb00fbdc57",
"triggerType" : "PUSH"
}, {
"hash" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29064",
"triggerID" : "dc499251534da1f28801ef639c8eb8ee261de3c8",
"triggerType" : "PUSH"
}, {
"hash" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cfb11b3de3e838c30e5c1a44edc919c775ee75fe",
"triggerType" : "PUSH"
}, {
"hash" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29105",
"triggerID" : "744b699a82b4640652050eb75eecaf78b2f10c2c",
"triggerType" : "PUSH"
}, {
"hash" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119",
"triggerID" : "e089768e55137235bd5b85a0b6b97b00dd0d23f9",
"triggerType" : "PUSH"
}, {
"hash" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "67a1b9f2f64179cd885df1385dff5fed43a58ec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cfb11b3de3e838c30e5c1a44edc919c775ee75fe UNKNOWN
* e089768e55137235bd5b85a0b6b97b00dd0d23f9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29119)
* 67a1b9f2f64179cd885df1385dff5fed43a58ec0 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 077423c8c94f59aade26c2c57001a4551a1b28af Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12d370196fdbe6ff098e8c93512b90fb774aa5c6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098)
* 50b889f3b887a3dafc99e8928f96c31b460b11a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153)
* e90077b322d2826456b7fbf4ba6f2c0c22f4c137 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "f3b9330d6489cb02148808b7c25a3eef48a8143e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f3b9330d6489cb02148808b7c25a3eef48a8143e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 077423c8c94f59aade26c2c57001a4551a1b28af Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971)
* f3b9330d6489cb02148808b7c25a3eef48a8143e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
slinkydeveloper commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r768823822
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
I wonder if this one can be a projectable decoding format as well, given it roughly uses the same code for deserializing of the other class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r768745796
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvCommons.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.ValidationException;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/** A class with common CSV format constants and utility methods. */
+public class CsvCommons {
Review comment:
Make `@Internal` and non-public.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormat.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
Review comment:
+Usage example
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormat.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
+ * @param <T>
+ */
+@PublicEvolving
+public class CsvFormat<T> extends SimpleStreamFormat<T> {
+
+ private static final long serialVersionUID = 1L;
+
+ private final CsvMapper mapper;
+ private final CsvSchema schema;
+ private final Class<Object> rootType;
+ private final Converter<Object, T, Void> converter;
+ private final TypeInformation<T> typeInformation;
+
+ @SuppressWarnings("unchecked")
+ <R> CsvFormat(
+ CsvMapper mapper,
+ CsvSchema schema,
+ Class<R> rootType,
+ Converter<R, T, Void> converter,
+ TypeInformation<T> typeInformation) {
+ this.mapper = checkNotNull(mapper);
+ this.schema = schema;
+ this.rootType = (Class<Object>) checkNotNull(rootType);
+ this.typeInformation = checkNotNull(typeInformation);
+ this.converter = (Converter<Object, T, Void>) checkNotNull(converter);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} using a {@code CsvSchema}.
+ *
+ * @param schema The Jackson CSV schema configured for parsing specific CSV files.
+ * @param typeInformation The Flink type descriptor of the returned elements.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(CsvSchema schema, TypeInformation<T> typeInformation) {
+ return from(new CsvMapper(), schema, typeInformation);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} using a {@code CsvSchema} and a pre-created {@code CsvMapper}.
+ *
+ * @param mapper The pre-created {@code CsvMapper}.
+ * @param schema The Jackson CSV schema configured for parsing specific CSV files.
+ * @param typeInformation The Flink type descriptor of the returned elements.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(
Review comment:
Rename to `forSchema`.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormat.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
+ * @param <T>
+ */
+@PublicEvolving
+public class CsvFormat<T> extends SimpleStreamFormat<T> {
+
+ private static final long serialVersionUID = 1L;
+
+ private final CsvMapper mapper;
+ private final CsvSchema schema;
+ private final Class<Object> rootType;
+ private final Converter<Object, T, Void> converter;
+ private final TypeInformation<T> typeInformation;
+
+ @SuppressWarnings("unchecked")
+ <R> CsvFormat(
+ CsvMapper mapper,
+ CsvSchema schema,
+ Class<R> rootType,
+ Converter<R, T, Void> converter,
+ TypeInformation<T> typeInformation) {
+ this.mapper = checkNotNull(mapper);
+ this.schema = schema;
+ this.rootType = (Class<Object>) checkNotNull(rootType);
+ this.typeInformation = checkNotNull(typeInformation);
+ this.converter = (Converter<Object, T, Void>) checkNotNull(converter);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} using a {@code CsvSchema}.
+ *
+ * @param schema The Jackson CSV schema configured for parsing specific CSV files.
+ * @param typeInformation The Flink type descriptor of the returned elements.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(CsvSchema schema, TypeInformation<T> typeInformation) {
+ return from(new CsvMapper(), schema, typeInformation);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} using a {@code CsvSchema} and a pre-created {@code CsvMapper}.
+ *
+ * @param mapper The pre-created {@code CsvMapper}.
+ * @param schema The Jackson CSV schema configured for parsing specific CSV files.
+ * @param typeInformation The Flink type descriptor of the returned elements.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(
+ CsvMapper mapper, CsvSchema schema, TypeInformation<T> typeInformation) {
+ return new CsvFormat<>(
+ mapper,
+ schema,
+ typeInformation.getTypeClass(),
+ (value, context) -> value,
+ typeInformation);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} for reading CSV files mapped to the provided POJO class
+ * definition.
+ *
+ * @param pojoType The type class of the POJO.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(Class<T> pojoType) {
+ CsvMapper mapper = new CsvMapper();
+ return from(mapper, mapper.schemaFor(pojoType), TypeInformation.of(pojoType));
+ }
+
+ @Override
+ public StreamFormat.Reader<T> createReader(Configuration config, FSDataInputStream stream)
+ throws IOException {
+ return new Reader<>(mapper.readerFor(rootType).with(schema).readValues(stream), converter);
+ }
+
+ @Override
+ public TypeInformation<T> getProducedType() {
+ return typeInformation;
+ }
+
+ // ------------------------------------------------------------------------
+
+ /** The actual reader for the {@code CsvFormat}. */
+ public static final class Reader<R, T> implements StreamFormat.Reader<T> {
Review comment:
private
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.formats.csv.RowDataToCsvConverters.RowDataToCsvConverter.RowDataToCsvFormatConverterContext;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectWriter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import java.io.IOException;
+
+/** A simple {@link BulkWriter} implementation based on Jackson CSV transformations. */
+@PublicEvolving
+public class CsvBulkWriter<T> implements BulkWriter<T> {
+
+ private final FSDataOutputStream stream;
+ private final CsvMapper mapper;
+ private final Converter<T, JsonNode, RowDataToCsvFormatConverterContext> converter;
+ private final ObjectWriter csvWriter;
+ /** Reusable within the converter. * */
+ private final transient ObjectNode container;
+
+ /**
+ * Constructs a writer with Jackson schema and a type converter.
+ *
+ * @param mapper The specialized mapper for producing CSV.
+ * @param schema The schema that defined the mapping properties.
+ * @param converter The type converter that converts incoming elements of type {@code <T>} into
+ * elements of type JsonNode.
+ * @param stream The output stream.
+ */
+ public CsvBulkWriter(
Review comment:
Make package-private and add factory methods.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormat.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
+ * @param <T>
+ */
+@PublicEvolving
+public class CsvFormat<T> extends SimpleStreamFormat<T> {
+
+ private static final long serialVersionUID = 1L;
+
+ private final CsvMapper mapper;
+ private final CsvSchema schema;
+ private final Class<Object> rootType;
+ private final Converter<Object, T, Void> converter;
+ private final TypeInformation<T> typeInformation;
+
+ @SuppressWarnings("unchecked")
+ <R> CsvFormat(
+ CsvMapper mapper,
+ CsvSchema schema,
+ Class<R> rootType,
+ Converter<R, T, Void> converter,
+ TypeInformation<T> typeInformation) {
+ this.mapper = checkNotNull(mapper);
+ this.schema = schema;
+ this.rootType = (Class<Object>) checkNotNull(rootType);
+ this.typeInformation = checkNotNull(typeInformation);
+ this.converter = (Converter<Object, T, Void>) checkNotNull(converter);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} using a {@code CsvSchema}.
+ *
+ * @param schema The Jackson CSV schema configured for parsing specific CSV files.
+ * @param typeInformation The Flink type descriptor of the returned elements.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(CsvSchema schema, TypeInformation<T> typeInformation) {
Review comment:
Rename to `forSchema`.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormat.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
+ * @param <T>
+ */
+@PublicEvolving
+public class CsvFormat<T> extends SimpleStreamFormat<T> {
Review comment:
I think we should go for a name that shows that this is the reading side.
Maybe `CsvReaderFormat`?
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormat.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
+ * @param <T>
Review comment:
+ doc on type parameter.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormat.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
+ * @param <T>
+ */
+@PublicEvolving
+public class CsvFormat<T> extends SimpleStreamFormat<T> {
+
+ private static final long serialVersionUID = 1L;
+
+ private final CsvMapper mapper;
+ private final CsvSchema schema;
+ private final Class<Object> rootType;
+ private final Converter<Object, T, Void> converter;
+ private final TypeInformation<T> typeInformation;
+
+ @SuppressWarnings("unchecked")
+ <R> CsvFormat(
+ CsvMapper mapper,
+ CsvSchema schema,
+ Class<R> rootType,
+ Converter<R, T, Void> converter,
+ TypeInformation<T> typeInformation) {
+ this.mapper = checkNotNull(mapper);
+ this.schema = schema;
+ this.rootType = (Class<Object>) checkNotNull(rootType);
+ this.typeInformation = checkNotNull(typeInformation);
+ this.converter = (Converter<Object, T, Void>) checkNotNull(converter);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} using a {@code CsvSchema}.
+ *
+ * @param schema The Jackson CSV schema configured for parsing specific CSV files.
+ * @param typeInformation The Flink type descriptor of the returned elements.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(CsvSchema schema, TypeInformation<T> typeInformation) {
+ return from(new CsvMapper(), schema, typeInformation);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} using a {@code CsvSchema} and a pre-created {@code CsvMapper}.
+ *
+ * @param mapper The pre-created {@code CsvMapper}.
+ * @param schema The Jackson CSV schema configured for parsing specific CSV files.
+ * @param typeInformation The Flink type descriptor of the returned elements.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(
+ CsvMapper mapper, CsvSchema schema, TypeInformation<T> typeInformation) {
+ return new CsvFormat<>(
+ mapper,
+ schema,
+ typeInformation.getTypeClass(),
+ (value, context) -> value,
+ typeInformation);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} for reading CSV files mapped to the provided POJO class
+ * definition.
+ *
+ * @param pojoType The type class of the POJO.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(Class<T> pojoType) {
Review comment:
Rename to `forPojo`.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##########
@@ -19,103 +19,48 @@
package org.apache.flink.formats.csv;
import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.common.serialization.SerializationSchema;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.configuration.ReadableConfig;
-import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.impl.StreamFormatAdapter;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.formats.common.Converter;
import org.apache.flink.table.connector.ChangelogMode;
-import org.apache.flink.table.connector.format.DecodingFormat;
-import org.apache.flink.table.connector.format.EncodingFormat;
-import org.apache.flink.table.connector.sink.DynamicTableSink;
-import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.Projection;
+import org.apache.flink.table.connector.format.BulkDecodingFormat;
+import org.apache.flink.table.connector.format.ProjectableDecodingFormat;
+import org.apache.flink.table.connector.source.DynamicTableSource.Context;
import org.apache.flink.table.data.RowData;
-import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.BulkReaderFormatFactory;
import org.apache.flink.table.factories.DynamicTableFactory;
-import org.apache.flink.table.factories.FactoryUtil;
-import org.apache.flink.table.factories.SerializationFormatFactory;
import org.apache.flink.table.types.DataType;
import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
import org.apache.commons.lang3.StringEscapeUtils;
import java.util.Collections;
-import java.util.HashSet;
import java.util.Set;
import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
-import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
-/**
- * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
- * and {@link DeserializationSchema}.
- */
+/** CSV format factory for file system. */
+//TODO: Rename back to CsvFileSystemFormatFactory? @Arvid
Review comment:
Why do you think so?
Can we find a name that doesn't start with Csv so it pops up later in auto-complete? `TableCsv...` or `InternalCsv...`?
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvToRowDataConverters.java
##########
@@ -305,7 +311,7 @@ private CsvToRowDataConverter createArrayConverter(ArrayType arrayType) {
}
private static void validateArity(int expected, int actual, boolean ignoreParseErrors) {
- if (expected != actual && !ignoreParseErrors) {
+ if (expected > actual && !ignoreParseErrors) {
Review comment:
Can you double-check if this is still needed?
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.formats.common.Converter;
+import org.apache.flink.formats.csv.RowDataToCsvConverters.RowDataToCsvConverter.RowDataToCsvFormatConverterContext;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectWriter;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import java.io.IOException;
+
+/** A simple {@link BulkWriter} implementation based on Jackson CSV transformations. */
+@PublicEvolving
+public class CsvBulkWriter<T> implements BulkWriter<T> {
+
+ private final FSDataOutputStream stream;
+ private final CsvMapper mapper;
+ private final Converter<T, JsonNode, RowDataToCsvFormatConverterContext> converter;
+ private final ObjectWriter csvWriter;
+ /** Reusable within the converter. * */
+ private final transient ObjectNode container;
Review comment:
Drop transient.
##########
File path: flink-formats/flink-csv/src/test/resources/META-INF/services/org.apache.flink.table.factories.Factory
##########
@@ -0,0 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+org.apache.flink.formats.csv.CsvFormatFactory
+org.apache.flink.formats.csv.CsvSerDeSchemaFactory
Review comment:
This should also be in earlier commits.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvToRowDataConverters.java
##########
@@ -65,7 +66,12 @@ public CsvToRowDataConverters(boolean ignoreParseErrors) {
* data structures.
*/
@FunctionalInterface
- public interface CsvToRowDataConverter extends Serializable {
+ public interface CsvToRowDataConverter extends Converter<JsonNode, Object, Void> {
Review comment:
Make sure to remove as much `public` as possible from non-public stuff.
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.unixTimestampToLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class CsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
+ public void testProjectPushDown() throws Exception {
Review comment:
Here there are no DataStream tests whatsoever. Please add them to cover your new functionality (don't bother covering old stuff).
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.unixTimestampToLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class CsvFormatITCase extends JsonPlanTestBase {
Review comment:
Add these tests to the respective production commits. If it's only reader, then into the first commit. If it's writer or both, then second commit.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormat.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.connector.file.src.reader.StreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.formats.common.Converter;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.MappingIterator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@code StreamFormat} for reading CSV files.
+ *
+ * @param <T>
+ */
+@PublicEvolving
+public class CsvFormat<T> extends SimpleStreamFormat<T> {
+
+ private static final long serialVersionUID = 1L;
+
+ private final CsvMapper mapper;
+ private final CsvSchema schema;
+ private final Class<Object> rootType;
+ private final Converter<Object, T, Void> converter;
+ private final TypeInformation<T> typeInformation;
+
+ @SuppressWarnings("unchecked")
+ <R> CsvFormat(
+ CsvMapper mapper,
+ CsvSchema schema,
+ Class<R> rootType,
+ Converter<R, T, Void> converter,
+ TypeInformation<T> typeInformation) {
+ this.mapper = checkNotNull(mapper);
+ this.schema = schema;
+ this.rootType = (Class<Object>) checkNotNull(rootType);
+ this.typeInformation = checkNotNull(typeInformation);
+ this.converter = (Converter<Object, T, Void>) checkNotNull(converter);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} using a {@code CsvSchema}.
+ *
+ * @param schema The Jackson CSV schema configured for parsing specific CSV files.
+ * @param typeInformation The Flink type descriptor of the returned elements.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(CsvSchema schema, TypeInformation<T> typeInformation) {
+ return from(new CsvMapper(), schema, typeInformation);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} using a {@code CsvSchema} and a pre-created {@code CsvMapper}.
+ *
+ * @param mapper The pre-created {@code CsvMapper}.
+ * @param schema The Jackson CSV schema configured for parsing specific CSV files.
+ * @param typeInformation The Flink type descriptor of the returned elements.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(
+ CsvMapper mapper, CsvSchema schema, TypeInformation<T> typeInformation) {
+ return new CsvFormat<>(
+ mapper,
+ schema,
+ typeInformation.getTypeClass(),
+ (value, context) -> value,
+ typeInformation);
+ }
+
+ /**
+ * Builds a new {@code CsvFormat} for reading CSV files mapped to the provided POJO class
+ * definition.
+ *
+ * @param pojoType The type class of the POJO.
+ * @param <T> The type of the returned elements.
+ */
+ public static <T> CsvFormat<T> from(Class<T> pojoType) {
Review comment:
Make sure this method is covered by a test and feels natural to DataStream/java dev.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770696351
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##########
@@ -19,103 +19,48 @@
package org.apache.flink.formats.csv;
import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.common.serialization.SerializationSchema;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.configuration.ReadableConfig;
-import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.impl.StreamFormatAdapter;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.formats.common.Converter;
import org.apache.flink.table.connector.ChangelogMode;
-import org.apache.flink.table.connector.format.DecodingFormat;
-import org.apache.flink.table.connector.format.EncodingFormat;
-import org.apache.flink.table.connector.sink.DynamicTableSink;
-import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.Projection;
+import org.apache.flink.table.connector.format.BulkDecodingFormat;
+import org.apache.flink.table.connector.format.ProjectableDecodingFormat;
+import org.apache.flink.table.connector.source.DynamicTableSource.Context;
import org.apache.flink.table.data.RowData;
-import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.BulkReaderFormatFactory;
import org.apache.flink.table.factories.DynamicTableFactory;
-import org.apache.flink.table.factories.FactoryUtil;
-import org.apache.flink.table.factories.SerializationFormatFactory;
import org.apache.flink.table.types.DataType;
import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
import org.apache.commons.lang3.StringEscapeUtils;
import java.util.Collections;
-import java.util.HashSet;
import java.util.Set;
import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
-import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
-/**
- * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
- * and {@link DeserializationSchema}.
- */
+/** CSV format factory for file system. */
+//TODO: Rename back to CsvFileSystemFormatFactory? @Arvid
Review comment:
I guess we could pass the `OutputStream` instead of the `FSDataOutputStream`, but that would mean adding a check via `isinstanceof` in the `finish()` method for calling `sync()` for files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770781617
##########
File path: flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatITCase.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.formats.common.TimeFormats;
+import org.apache.flink.table.planner.runtime.utils.TestData;
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.planner.utils.JsonPlanTestBase;
+
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.table.utils.DateTimeUtils.unixTimestampToLocalDateTime;
+
+/** Tests for the CSV file format. */
+public class CsvFormatITCase extends JsonPlanTestBase {
+
+ @Test
+ public void testProjectPushDown() throws Exception {
+ List<String> data = Arrays.asList("1,1,hi", "2,1,hello", "3,2,hello world");
+ createSourceTable("MyTable", data, "a bigint", "b int not null", "c varchar");
+ File sinkPath = createSinkTable("MySink", "a bigint", "c varchar");
+
+ String jsonPlan = tableEnv.getJsonPlan("insert into MySink select a, c from MyTable");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("1,hi", "2,hello", "3,hello world"), sinkPath);
+ }
+
+ @Test
+ public void testReadingMetadata() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.smallData3()),
+ new String[] {"a int", "b bigint", "m varchar metadata"},
+ new HashMap<String, String>() {
+ {
+ put("readable-metadata", "m:STRING");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a bigint", "m varchar");
+
+ String jsonPlan = tableEnv.getJsonPlan("insert into MySink select a, m from MyTable");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("1,Hi", "2,Hello", "3,Hello world"), sinkPath);
+ }
+
+ @Test
+ public void testFilterPushDown() throws Exception {
+ List<String> data = Arrays.asList("1,1,hi", "2,1,hello", "3,2,hello world");
+ createSourceTable("MyTable", data, "a bigint", "b int not null", "c varchar");
+ File sinkPath = createSinkTable("MySink", "a bigint", "b int", "c varchar");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select * from MyTable where a > 1");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("2,1,hello", "3,2,hello world"), sinkPath);
+ }
+
+ @Test
+ public void testPartitionPushDown() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.smallData3()),
+ new String[] {"a int", "p bigint", "c varchar"},
+ "p",
+ new HashMap<String, String>() {
+ {
+ put("partition-list", "p:1;p:2");
+ }
+ });
+ File sinkPath = createSinkTable("MySink", "a int", "p bigint", "c varchar");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select * from MyTable where p = 2");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(Arrays.asList("2,2,Hello", "3,2,Hello world"), sinkPath);
+ }
+
+ @Test
+ public void testWatermarkPushDown() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.data3WithTimestamp()),
+ new String[] {
+ "a int",
+ "b bigint",
+ "c varchar",
+ "ts timestamp(3)",
+ "watermark for ts as ts - interval '5' second"
+ },
+ new HashMap<String, String>() {
+ {
+ put("enable-watermark-push-down", "true");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a int", "b bigint", "ts timestamp(3)");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan("insert into MySink select a, b, ts from MyTable where b = 3");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(
+ Arrays.asList(
+ "4,3," + formatSqlTimestamp(4000L),
+ "5,3," + formatSqlTimestamp(5000L),
+ "6,3," + formatSqlTimestamp(6000L)),
+ sinkPath);
+ }
+
+ @Test
+ public void testPushDowns() throws Exception {
+ createTestValuesSourceTable(
+ "MyTable",
+ JavaScalaConversionUtil.toJava(TestData.data3WithTimestamp()),
+ new String[] {
+ "a int",
+ "b bigint",
+ "c varchar",
+ "ts timestamp(3)",
+ "watermark for ts as ts - interval '5' second"
+ },
+ "b",
+ new HashMap<String, String>() {
+ {
+ put("readable-metadata", "a:INT");
+ put("filterable-fields", "a");
+ put("enable-watermark-push-down", "true");
+ put("partition-list", "b:1;b:2;b:3;b:4;b:5;b:6");
+ }
+ });
+
+ File sinkPath = createSinkTable("MySink", "a int", "ts timestamp(3)");
+
+ String jsonPlan =
+ tableEnv.getJsonPlan(
+ "insert into MySink select a, ts from MyTable where b = 3 and a > 4");
+ tableEnv.executeJsonPlan(jsonPlan).await();
+
+ assertResult(
+ Arrays.asList("5," + formatSqlTimestamp(5000L), "6," + formatSqlTimestamp(6000L)),
+ sinkPath);
+ }
+
+ private static String formatSqlTimestamp(long timestamp) {
+ return TimeFormats.SQL_TIMESTAMP_FORMAT.format(unixTimestampToLocalDateTime(timestamp));
+ }
Review comment:
I will skip this one - the test data from the SQL suite is created based on local datetime, so the tests break if another zone is used anyhow: https://github.com/apache/flink/blob/a7192af8707f3f0d0f30fc71f3477edd92135cac/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/utils/TestData.scala#L267
I do not see it as an issue, because
1) No absolut values are used - only relative since 1970 for both the test data and the expected results.
2) The local time zone will this way always be used and I do not see how it can break on another machine because it using a different timezone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "6f95051fa8153ff7745b479e75e44e9b0ea496f0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6f95051fa8153ff7745b479e75e44e9b0ea496f0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 29e6025b084eb1c850ab3901c757dcfcb442c67e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289)
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
* 6f95051fa8153ff7745b479e75e44e9b0ea496f0 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
slinkydeveloper commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r771554155
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
> Is this fully supported in Table-land? If so, then I agree we should change it. It's a good performance-improvement.
Yes, please check the documentation of `DecodingFormat` which goes through all the implementation details both for formats and connectors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "6f95051fa8153ff7745b479e75e44e9b0ea496f0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6f95051fa8153ff7745b479e75e44e9b0ea496f0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
* 6f95051fa8153ff7745b479e75e44e9b0ea496f0 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r770679363
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##########
@@ -19,103 +19,48 @@
package org.apache.flink.formats.csv;
import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.common.serialization.SerializationSchema;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.configuration.ReadableConfig;
-import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.impl.StreamFormatAdapter;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.formats.common.Converter;
import org.apache.flink.table.connector.ChangelogMode;
-import org.apache.flink.table.connector.format.DecodingFormat;
-import org.apache.flink.table.connector.format.EncodingFormat;
-import org.apache.flink.table.connector.sink.DynamicTableSink;
-import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.Projection;
+import org.apache.flink.table.connector.format.BulkDecodingFormat;
+import org.apache.flink.table.connector.format.ProjectableDecodingFormat;
+import org.apache.flink.table.connector.source.DynamicTableSource.Context;
import org.apache.flink.table.data.RowData;
-import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.BulkReaderFormatFactory;
import org.apache.flink.table.factories.DynamicTableFactory;
-import org.apache.flink.table.factories.FactoryUtil;
-import org.apache.flink.table.factories.SerializationFormatFactory;
import org.apache.flink.table.types.DataType;
import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
import org.apache.commons.lang3.StringEscapeUtils;
import java.util.Collections;
-import java.util.HashSet;
import java.util.Set;
import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
-import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
-/**
- * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
- * and {@link DeserializationSchema}.
- */
+/** CSV format factory for file system. */
+//TODO: Rename back to CsvFileSystemFormatFactory? @Arvid
Review comment:
I had this thought because `BulkReader` is a `flink-connector-file` interface, and `CsvBulkWriter` writes into `FSDataOutputStream`. Both the encoder and the decoder exposed by the factory are for files.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##########
@@ -19,103 +19,48 @@
package org.apache.flink.formats.csv;
import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.common.serialization.SerializationSchema;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.configuration.ReadableConfig;
-import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.impl.StreamFormatAdapter;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.formats.common.Converter;
import org.apache.flink.table.connector.ChangelogMode;
-import org.apache.flink.table.connector.format.DecodingFormat;
-import org.apache.flink.table.connector.format.EncodingFormat;
-import org.apache.flink.table.connector.sink.DynamicTableSink;
-import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.Projection;
+import org.apache.flink.table.connector.format.BulkDecodingFormat;
+import org.apache.flink.table.connector.format.ProjectableDecodingFormat;
+import org.apache.flink.table.connector.source.DynamicTableSource.Context;
import org.apache.flink.table.data.RowData;
-import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.BulkReaderFormatFactory;
import org.apache.flink.table.factories.DynamicTableFactory;
-import org.apache.flink.table.factories.FactoryUtil;
-import org.apache.flink.table.factories.SerializationFormatFactory;
import org.apache.flink.table.types.DataType;
import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
import org.apache.commons.lang3.StringEscapeUtils;
import java.util.Collections;
-import java.util.HashSet;
import java.util.Set;
import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
-import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
-/**
- * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
- * and {@link DeserializationSchema}.
- */
+/** CSV format factory for file system. */
+//TODO: Rename back to CsvFileSystemFormatFactory? @Arvid
Review comment:
I like the `TableCsvFormatFactory` - will rename it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r771408712
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##########
@@ -19,103 +19,48 @@
package org.apache.flink.formats.csv;
import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.serialization.DeserializationSchema;
-import org.apache.flink.api.common.serialization.SerializationSchema;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.configuration.ReadableConfig;
-import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.impl.StreamFormatAdapter;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.formats.common.Converter;
import org.apache.flink.table.connector.ChangelogMode;
-import org.apache.flink.table.connector.format.DecodingFormat;
-import org.apache.flink.table.connector.format.EncodingFormat;
-import org.apache.flink.table.connector.sink.DynamicTableSink;
-import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.Projection;
+import org.apache.flink.table.connector.format.BulkDecodingFormat;
+import org.apache.flink.table.connector.format.ProjectableDecodingFormat;
+import org.apache.flink.table.connector.source.DynamicTableSource.Context;
import org.apache.flink.table.data.RowData;
-import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.BulkReaderFormatFactory;
import org.apache.flink.table.factories.DynamicTableFactory;
-import org.apache.flink.table.factories.FactoryUtil;
-import org.apache.flink.table.factories.SerializationFormatFactory;
import org.apache.flink.table.types.DataType;
import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
import org.apache.commons.lang3.StringEscapeUtils;
import java.util.Collections;
-import java.util.HashSet;
import java.util.Set;
import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
-import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
-/**
- * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
- * and {@link DeserializationSchema}.
- */
+/** CSV format factory for file system. */
+//TODO: Rename back to CsvFileSystemFormatFactory? @Arvid
Review comment:
I forgot that the `BulkWriter` factory is passed an `FSDataOutputStream`, so technically both ins and outs of this factory are only for supporting files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
}, {
"hash" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26045",
"triggerID" : "47b6d1194869dc00a6d326bf87127da8c6d667a3",
"triggerType" : "PUSH"
}, {
"hash" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26192",
"triggerID" : "6aea9ddf28bf11ed35c87ef57e477fecca94f266",
"triggerType" : "PUSH"
}, {
"hash" : "4e5445170476fae8081037143c62c36318677ad6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26364",
"triggerID" : "4e5445170476fae8081037143c62c36318677ad6",
"triggerType" : "PUSH"
}, {
"hash" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26391",
"triggerID" : "0c1c04317c7fb3f4470cdf596756172105d25cda",
"triggerType" : "PUSH"
}, {
"hash" : "7e097463582d225c33b56c3d2f067a3954867a11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26599",
"triggerID" : "7e097463582d225c33b56c3d2f067a3954867a11",
"triggerType" : "PUSH"
}, {
"hash" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26971",
"triggerID" : "077423c8c94f59aade26c2c57001a4551a1b28af",
"triggerType" : "PUSH"
}, {
"hash" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27098",
"triggerID" : "12d370196fdbe6ff098e8c93512b90fb774aa5c6",
"triggerType" : "PUSH"
}, {
"hash" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27153",
"triggerID" : "50b889f3b887a3dafc99e8928f96c31b460b11a8",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"triggerType" : "PUSH"
}, {
"hash" : "e90077b322d2826456b7fbf4ba6f2c0c22f4c137",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27155",
"triggerID" : "986209712",
"triggerType" : "MANUAL"
}, {
"hash" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28289",
"triggerID" : "29e6025b084eb1c850ab3901c757dcfcb442c67e",
"triggerType" : "PUSH"
}, {
"hash" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331",
"triggerID" : "8074cb4ace3b77d71d67926d289d7322ff3eb4c2",
"triggerType" : "PUSH"
}, {
"hash" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336",
"triggerID" : "bea711eea2b6af329e8b49c3fdd08ee74f90813f",
"triggerType" : "PUSH"
}, {
"hash" : "949b964bcb65db05a9c903a28f59f933300a7935",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "949b964bcb65db05a9c903a28f59f933300a7935",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8074cb4ace3b77d71d67926d289d7322ff3eb4c2 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28331)
* bea711eea2b6af329e8b49c3fdd08ee74f90813f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28336)
* 949b964bcb65db05a9c903a28f59f933300a7935 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17598:
URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25611",
"triggerID" : "9aa8115b9b3fa8cc853f7821c29f939870c79e5d",
"triggerType" : "PUSH"
}, {
"hash" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635",
"triggerID" : "ee8fa611b135924ad3a083ff71dbcbc2e1895351",
"triggerType" : "PUSH"
}, {
"hash" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687",
"triggerID" : "0e3bc85342e7ae33117597931baa83f7c78c19a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ee8fa611b135924ad3a083ff71dbcbc2e1895351 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635)
* 0e3bc85342e7ae33117597931baa83f7c78c19a4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25687)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] afedulov commented on a change in pull request #17598: [FLINK-24703][connectors][formats] Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
Posted by GitBox <gi...@apache.org>.
afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r773800347
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
@slinkydeveloper
I am not sure about this actually. The focus of this PR is
_Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces._
I would rather not touch the existing Ser/De schema-based functionality in this scope. Nothing will break if we won't support pushdown in the non-filesystem connectors for now and to me, it sounds like an improvement that should be done in a separate PR.
##########
File path: flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvSerDeSchemaFactory.java
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+
+import java.util.Collections;
+import java.util.Set;
+
+import static org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER;
+import static org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS;
+import static org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL;
+import static org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+@Internal
+public final class CsvSerDeSchemaFactory
+ implements DeserializationFormatFactory, SerializationFormatFactory {
+
+ @Override
+ public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+ DynamicTableFactory.Context context, ReadableConfig formatOptions) {
+ FactoryUtil.validateFactoryOptions(this, formatOptions);
+ CsvCommons.validateFormatOptions(formatOptions);
+
+ return new DecodingFormat<DeserializationSchema<RowData>>() {
+ @Override
+ public DeserializationSchema<RowData> createRuntimeDecoder(
Review comment:
@slinkydeveloper
I am not sure about this actually. The focus of this PR is
_Add CSV format support for filesystem **based on StreamFormat and BulkWriter interfaces**._
I would rather not touch the existing Ser/De schema-based functionality in this scope. Nothing will break if we won't support pushdown in the non-filesystem connectors for now and to me, it sounds like an improvement that should be done in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org