You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/03/30 21:36:25 UTC

[GitHub] [flink] afedulov opened a new pull request #19286: [FLINK-25931] Add projection pushdown support for CsvFormatFactory

afedulov opened a new pull request #19286:
URL: https://github.com/apache/flink/pull/19286


   ## What is the purpose of the change
   FLINK-24703 added support for projection pushdown in CsvFileFormatFactory. This PR adds the same functionality for non-file-based connectors that use CsvFormatFactory and on Ser/DeSerialization schemas.
   
   ## Brief change log
   
    - Switches CsvFormatFactory to using ProjectableDecodingFormat instead of simple DecodingFormat
    - A distinction is made between the physical(read from file) RowType and the returned projected RowData type
    - CsvRowDataDeserializationSchema is initialized with both physical and projected RowTypes and filters out unused fields from the returned results
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
     - Adds and a unit test that verifies that the fields not used in the TableAPI are filtered out
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / *no*)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / *no*)
     - The serializers: (*yes* / no / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / *no* / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / *no* / don't know)
     - The S3 file system connector: (yes / *no* / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (*yes* / no)
     - If yes, how is the feature documented? (*not applicable - internal optimization* / docs / JavaDocs / not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #19286: [FLINK-25931] Add projection pushdown support for CsvFormatFactory

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19286:
URL: https://github.com/apache/flink/pull/19286#issuecomment-1083653731


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ab7e810e56b04ecb49ac8c82360d56fc78aee584",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33991",
       "triggerID" : "ab7e810e56b04ecb49ac8c82360d56fc78aee584",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "1084422541",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "6ec4b0b242c099430437047c449a1ee3402ba0dc",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34036",
       "triggerID" : "1084422541",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "6ec4b0b242c099430437047c449a1ee3402ba0dc",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34036",
       "triggerID" : "6ec4b0b242c099430437047c449a1ee3402ba0dc",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   *  Unknown: [CANCELED](TBD) 
   * 6ec4b0b242c099430437047c449a1ee3402ba0dc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34036) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] afedulov commented on pull request #19286: [FLINK-25931] Add projection pushdown support for CsvFormatFactory

Posted by GitBox <gi...@apache.org>.
afedulov commented on pull request #19286:
URL: https://github.com/apache/flink/pull/19286#issuecomment-1084422541


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #19286: [FLINK-25931] Add projection pushdown support for CsvFormatFactory

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19286:
URL: https://github.com/apache/flink/pull/19286#issuecomment-1083653731


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ab7e810e56b04ecb49ac8c82360d56fc78aee584",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33991",
       "triggerID" : "ab7e810e56b04ecb49ac8c82360d56fc78aee584",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ec4b0b242c099430437047c449a1ee3402ba0dc",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1084422541",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "1084422541",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "6ec4b0b242c099430437047c449a1ee3402ba0dc",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6ec4b0b242c099430437047c449a1ee3402ba0dc",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   *  Unknown: [CANCELED](TBD) 
   * 6ec4b0b242c099430437047c449a1ee3402ba0dc UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #19286: [FLINK-25931] Add projection pushdown support for CsvFormatFactory

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19286:
URL: https://github.com/apache/flink/pull/19286#issuecomment-1083653731


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ab7e810e56b04ecb49ac8c82360d56fc78aee584",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33991",
       "triggerID" : "ab7e810e56b04ecb49ac8c82360d56fc78aee584",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ec4b0b242c099430437047c449a1ee3402ba0dc",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34036",
       "triggerID" : "1084422541",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "6ec4b0b242c099430437047c449a1ee3402ba0dc",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34036",
       "triggerID" : "6ec4b0b242c099430437047c449a1ee3402ba0dc",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "1084422541",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 6ec4b0b242c099430437047c449a1ee3402ba0dc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=34036) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #19286: [FLINK-25931] Add projection pushdown support for CsvFormatFactory

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19286:
URL: https://github.com/apache/flink/pull/19286#issuecomment-1083653731


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ab7e810e56b04ecb49ac8c82360d56fc78aee584",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33991",
       "triggerID" : "ab7e810e56b04ecb49ac8c82360d56fc78aee584",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ab7e810e56b04ecb49ac8c82360d56fc78aee584 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33991) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #19286: [FLINK-25931] Add projection pushdown support for CsvFormatFactory

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19286:
URL: https://github.com/apache/flink/pull/19286#issuecomment-1083653731


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ab7e810e56b04ecb49ac8c82360d56fc78aee584",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33991",
       "triggerID" : "ab7e810e56b04ecb49ac8c82360d56fc78aee584",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ab7e810e56b04ecb49ac8c82360d56fc78aee584 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33991) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #19286: [FLINK-25931] Add projection pushdown support for CsvFormatFactory

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #19286:
URL: https://github.com/apache/flink/pull/19286#issuecomment-1083653731


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ab7e810e56b04ecb49ac8c82360d56fc78aee584",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ab7e810e56b04ecb49ac8c82360d56fc78aee584",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ab7e810e56b04ecb49ac8c82360d56fc78aee584 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org