You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "BugsEverywhere (via GitHub)" <gi...@apache.org> on 2026/10/13 02:12:07 UTC

[PR] [FLINK-40471][python] Support per-UDF concurrency and batch size in DataFrame API [flink]

BugsEverywhere opened a new pull request, #29470:
URL: https://github.com/apache/flink/pull/29470

## What is the purpose of the change

Implement [FLINK-40471](https://issues.apache.org/jira/browse/FLINK-40471), as specified in FLIP-591. DataFrame scalar UDFs can configure their Python stage's parallelism with `concurrency` and pandas/Arrow batch sizes with `batch_size`, independently of other UDFs. Omitting either option preserves the existing behavior.

## Brief change log

- Validate the optional positive integer parameters and reject `batch_size` for general UDFs.
- Carry execution settings into Java Python functions, split UDFs with different settings into separate stages, and apply each stage's parallelism and batch-size overrides.
- Document the parameters and add declaration, planner, and execution tests.

## Verifying this change

- All 28 DataFrame UDF tests pass, covering validation, defaults, streaming/batch plans with separate and nested UDFs, and actual pandas/Arrow batch sizes in the same pipeline.
- Existing Python function-kind splitting and nested CSE Java tests pass (33 tests).
- Maven reactor verification on Java 21 completed across all modules over resumed runs.
- Spotless, Checkstyle, Python type checking, and changed-file Flake8 pass.
- Full Sphinx documentation build passes with warnings treated as errors.

## Does this pull request potentially affect one of the following parts:

- Dependencies: no.
- Public API: yes; optional DataFrame `udf()` parameters specified in FLIP-591.
- Serializers: no state serializer changes.
- Runtime per-record code paths: no; existing operators are configured during planning.
- Deployment or recovery: no changes to checkpointing or cluster deployment; explicitly configured Python stages use the requested parallelism.
- S3 file system connector: no.

## Documentation

- New feature: yes.
- Documented in the Python API docstring and DataFrame UDF Sphinx reference.

---

##### Was generative AI tooling used to co-author this PR?

- [X] Yes (please specify the tool below)

Generated-by: Codex (GPT-6)

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-40471][python] Support per-UDF concurrency and batch size in DataFrame API [flink]

Posted by "BugsEverywhere (via GitHub)" <gi...@apache.org>.

BugsEverywhere closed pull request #29470: [FLINK-40471][python] Support per-UDF concurrency and batch size in DataFrame API
URL: https://github.com/apache/flink/pull/29470


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-40471][python] Support per-UDF concurrency and batch size in DataFrame API [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.

flinkbot commented on PR #29470:
URL: https://github.com/apache/flink/pull/29470#issuecomment-6136818764

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a24a2bce6c793bf6ca87b2673fa38c7d0f9b3f16",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a24a2bce6c793bf6ca87b2673fa38c7d0f9b3f16",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a24a2bce6c793bf6ca87b2673fa38c7d0f9b3f16 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-40471][python] Support per-UDF concurrency and batch size in DataFrame API [flink]

Posted by "BugsEverywhere (via GitHub)" <gi...@apache.org>.

BugsEverywhere commented on PR #29470:
URL: https://github.com/apache/flink/pull/29470#issuecomment-6185481120

   Closing in favor of #29048, as agreed with Liu Liu in the JIRA discussion: https://issues.apache.org/jira/browse/FLINK-40471?focusedCommentId=18127350&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-18127350
   
   Thanks for reviewing this implementation. Please feel free to reuse any relevant implementation or tests from this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org