You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ferenc-csaky (via GitHub)" <gi...@apache.org> on 2025/01/21 13:24:45 UTC
[PR] [FLINK-37099][docs] Document memory/performance considerations and config possibilities for Netty4 [flink]
ferenc-csaky opened a new pull request, #26043:
URL: https://github.com/apache/flink/pull/26043
## What is the purpose of the change
Add some guidance in the Flink docs about Netty4 memory and performance configs.
## Verifying this change
Doc change
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37099][docs] Document memory/performance considerations and config possibilities for Netty4 [flink]
Posted by "ferenc-csaky (via GitHub)" <gi...@apache.org>.
ferenc-csaky commented on PR #26043:
URL: https://github.com/apache/flink/pull/26043#issuecomment-2604858330
> LGTM. I wonder if the netty 4 reflection setting should be enabled by default?
IMO it would still be safe top append it to `env.java.opts.all`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37099][docs] Document memory/performance considerations and config possibilities for Netty4 [flink]
Posted by "asardaes (via GitHub)" <gi...@apache.org>.
asardaes commented on PR #26043:
URL: https://github.com/apache/flink/pull/26043#issuecomment-2605438309
In my experience with Netty outside of Flink, running with the default (`io.netty.maxDirectMemory = -1`) means that the value of `-XX:MaxDirectMemorySize` will *not* be respected by Netty, so I think `taskmanager.memory.framework.off-heap.size` will no longer account for everything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37099][docs] Document memory/performance considerations and config possibilities for Netty4 [flink]
Posted by "ferenc-csaky (via GitHub)" <gi...@apache.org>.
ferenc-csaky commented on PR #26043:
URL: https://github.com/apache/flink/pull/26043#issuecomment-2605213796
> I asked about this in the mailing list but I think it got lost. Isn't `io.netty.maxDirectMemory` also important? Although that seems to only be documented [in the source code](https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java#L162).
I'm not sure if it would be a good idea to bring Netty-specific memory mgmt to the Flink docs. Personally I'd rather keep this doc part as light as it can be, to make sure we do not add any noise and create confusion where users might shoot themselves in the foot. Mixing that with JVM memory handling props from Flink configs could result in undesired setups pretty easily IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37099][docs] Document memory/performance considerations and config possibilities for Netty4 [flink]
Posted by "ferenc-csaky (via GitHub)" <gi...@apache.org>.
ferenc-csaky merged PR #26043:
URL: https://github.com/apache/flink/pull/26043
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37099][docs] Document memory/performance considerations and config possibilities for Netty4 [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26043:
URL: https://github.com/apache/flink/pull/26043#issuecomment-2605007372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3bea726773d9f287495b1eef7bdc0080b9203155",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3bea726773d9f287495b1eef7bdc0080b9203155",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3bea726773d9f287495b1eef7bdc0080b9203155 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37099][docs] Document memory/performance considerations and config possibilities for Netty4 [flink]
Posted by "asardaes (via GitHub)" <gi...@apache.org>.
asardaes commented on PR #26043:
URL: https://github.com/apache/flink/pull/26043#issuecomment-2605022296
I asked about this in the mailing list but I think it got lost. Isn't `io.netty.maxDirectMemory` also important? Although that seems to only be documented [in the source code](https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java#L162).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37099][docs] Document memory/performance considerations and config possibilities for Netty4 [flink]
Posted by "afedulov (via GitHub)" <gi...@apache.org>.
afedulov commented on PR #26043:
URL: https://github.com/apache/flink/pull/26043#issuecomment-2605293828
@asardaes I agree with @ferenc-csaky . People who are running in such memory-constrained scenarios are welcome to investigate the details of Netty configuration. The documentation section is there primarily to indicate what is the reason for the change in behavior inside of Flink. The path for a "normal" Flink user would be to simply allocate slightly more overall memory to TMs/JM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org