[jira] [Created] (FLINK-35695) Release Testing: Verify FLINK-32315: Support local file upload in K8s mode

["Ferenc Csaky (Jira)" <ji...@apache.org>]

Ferenc Csaky created FLINK-35695:
------------------------------------

             Summary: Release Testing: Verify FLINK-32315: Support local file upload in K8s mode
                 Key: FLINK-35695
                 URL: https://issues.apache.org/jira/browse/FLINK-35695
             Project: Flink
          Issue Type: Sub-task
          Components: Runtime / Network
            Reporter: Ferenc Csaky
             Fix For: 1.20.0


Follow up the test for https://issues.apache.org/jira/browse/FLINK-35533

In Flink 1.20,  we proposed integrating Flink's Hybrid Shuffle with Apache Celeborn through a pluggable remote tier interface. To verify this feature, you should reference these main two steps.

1. Implement Celeborn tier.
 * Implement a new tier factory and tier for Celeborn, including these APIs, including TierFactory/TierMasterAgent/TierProducerAgent/TierConsumerAgent.
 * The implementations should support granular data management at the Segment level for both client and server sides.

2. Use the implemented tier to shuffle data.
 * Compile Flink and Celeborn.
 * Deploy Celeborn service
 ** Deploy a new Celeborn service with the new compiled packages. You can reference the doc ([https://celeborn.apache.org/docs/latest/]) to deploy the cluster.
 * Add the compiled flink plugin jar (celeborn-client-flink-xxx.jar) to Flink classpath.
 * Configure the options to enable the feature.
 ** Configure the option taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class to the new Celeborn tier classes. Except for this option, the following options should also be added.

{code:java}
execution.batch-shuffle-mode: ALL_EXCHANGES_HYBRID_FULL 
celeborn.master.endpoints: <the celeborn endpoint address>
celeborn.client.shuffle.partition.type: MAP{code}
 * Run some test examples(e.g., WordCount) to verify the feature.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)