You are viewing a plain text version of this content. The canonical link for it is here.
- Re: [PR] [CELEBORN-894][FOLLOWUP] update commitMeta before update subPartition… [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/01 06:05:54 UTC, 1 replies.
- Re: [PR] [CELEBORN-2047] Support MapPartitionData on DFS [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/01 06:36:45 UTC, 6 replies.
- Re: [I] Who is using Apache Celeborn? [celeborn] - posted by "gaoyajun02 (via GitHub)" <gi...@apache.org> on 2025/07/01 08:03:30 UTC, 1 replies.
- Re: [PR] [CELEBORN-2016] Add cooldown time in worker shutdown [celeborn] - posted by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2025/07/01 08:38:24 UTC, 2 replies.
- Re: [PR] [CELEBORN-1768][SPARK-3][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn] - posted by "FMX (via GitHub)" <gi...@apache.org> on 2025/07/02 06:54:24 UTC, 1 replies.
- Re: [PR] [CELEBORN-2051] Celeborn support write map partition to HDFS [celeborn] - posted by "FMX (via GitHub)" <gi...@apache.org> on 2025/07/02 07:10:57 UTC, 0 replies.
- Re: [PR] CELEBORN-1991: add retries for registerShuffle on the lifecycleManager [celeborn] - posted by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2025/07/02 08:38:18 UTC, 1 replies.
- Re: [PR] [RFT] Enable run java UT (not a formal PR, just for discussion purposes) [celeborn] - posted by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2025/07/02 08:38:20 UTC, 1 replies.
- Re: [PR] [CELEBORN-1841][FOLLOWUP] Support custom implementation of EventExecutorChooser to avoid deadlock when calling await in EventLoop thread for EpollEventLoopGroup [celeborn] - posted by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2025/07/02 08:38:21 UTC, 5 replies.
- Re: [PR] [CELEBORN-894] Add checksum for shuffle data [celeborn] - posted by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2025/07/02 08:38:22 UTC, 1 replies.
- Re: [PR] [CELEBORN-2051] Support write MapPartition to DFS [celeborn] - posted by "FMX (via GitHub)" <gi...@apache.org> on 2025/07/03 02:51:39 UTC, 2 replies.
- Re: [PR] [CELEBORN-1984] Merge ResourceRequest to transportMessageProtobuf [celeborn] - posted by "FMX (via GitHub)" <gi...@apache.org> on 2025/07/03 06:02:12 UTC, 3 replies.
- Re: [PR] [CELEBORN-1917] supports celeborn.client.push.maxBytesSizeInFlight [celeborn] - posted by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2025/07/03 08:37:43 UTC, 0 replies.
- [PR] [CELEBORN-1844][Followup] alway try to use memory storage if available [celeborn] - posted by "FMX (via GitHub)" <gi...@apache.org> on 2025/07/03 14:32:11 UTC, 2 replies.
- [PR] refactor remote storage configration usage. [celeborn] - posted by "Kalvin2077 (via GitHub)" <gi...@apache.org> on 2025/07/04 09:19:44 UTC, 0 replies.
- [PR] [CELEBORN-2011][INFRA][FOLLOWUP] Correct author name [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/05 01:49:28 UTC, 2 replies.
- [PR] [CELEBORN-2054] simplify slow RPC logging to reduce verbosity [celeborn] - posted by "gaoyajun02 (via GitHub)" <gi...@apache.org> on 2025/07/06 00:03:10 UTC, 7 replies.
- Re: [PR] [CELEBORN-1903] Enable connection heartbeat and close idle connection [celeborn] - posted by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2025/07/06 08:36:38 UTC, 1 replies.
- [PR] [CELEBORN-2055] Fix some typos [celeborn] - posted by "codenohup (via GitHub)" <gi...@apache.org> on 2025/07/06 11:59:23 UTC, 3 replies.
- [PR] [CELEBORN-2052] Fix unexpected warning logs in Flink caused by duplicate BufferStreamEnd messages [celeborn] - posted by "codenohup (via GitHub)" <gi...@apache.org> on 2025/07/06 13:41:16 UTC, 2 replies.
- Re: [PR] [INFRA] More contributors name mapping [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/06 21:44:45 UTC, 1 replies.
- Re: [PR] [MINOR] Fixed some of the maven warnings [celeborn] - posted by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2025/07/07 08:39:16 UTC, 2 replies.
- [PR] [CELEBORN-2056] Make the wait time for the client to read non shuffle partitions configurable [celeborn] - posted by "dh20 (via GitHub)" <gi...@apache.org> on 2025/07/07 08:57:36 UTC, 13 replies.
- [PR] [CELEBORN-2057] Bump ap-loader version from 3.0-9 to 4.0-10 [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/07 13:05:25 UTC, 1 replies.
- Re: [PR] [CELEBORN-2031] Interruption Aware Slot Selection [celeborn] - posted by "akpatnam25 (via GitHub)" <gi...@apache.org> on 2025/07/07 17:25:31 UTC, 13 replies.
- [PR] [CELEBORN-2058] Add retry to avoid commtting failed when using HDFS storage [celeborn] - posted by "taowenjun (via GitHub)" <gi...@apache.org> on 2025/07/08 04:37:13 UTC, 0 replies.
- Re: [PR] [CIP-19][CELEBORN-1937] QuotaManager should support app threshold [celeborn] - posted by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2025/07/08 08:38:32 UTC, 2 replies.
- [PR] [CELEBORN-2060] Refactor `SlotsAlloctor` for better scalability [celeborn] - posted by "Kalvin2077 (via GitHub)" <gi...@apache.org> on 2025/07/08 09:14:17 UTC, 1 replies.
- Re: [PR] [CELEBORN-2058] Add retry to avoid commtting failed with HDFS storage [celeborn] - posted by "FMX (via GitHub)" <gi...@apache.org> on 2025/07/08 10:05:45 UTC, 6 replies.
- Re: [PR] [CELEBORN-2053] Refactor remote storage configration usage. [celeborn] - posted by "RexXiong (via GitHub)" <gi...@apache.org> on 2025/07/08 15:28:18 UTC, 9 replies.
- Re: [PR] [CIP-19][CELEBORN-1939] CongestionController should not limit user push speed when worker load is low [celeborn] - posted by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2025/07/09 08:38:32 UTC, 3 replies.
- [PR] [CELEBORN-2061] Introduce metrics to count the amount of data flushed into different storage types [celeborn] - posted by "TheodoreLx (via GitHub)" <gi...@apache.org> on 2025/07/09 13:39:41 UTC, 6 replies.
- Re: [PR] [CELEBORN-2032] Create reader should change to peer by taskAttemptId [celeborn] - posted by "leixm (via GitHub)" <gi...@apache.org> on 2025/07/10 04:08:39 UTC, 0 replies.
- [PR] [CELEBORN-2062] Introduce splitNum to optimize shuffle write [celeborn] - posted by "littlexyw (via GitHub)" <gi...@apache.org> on 2025/07/11 09:14:00 UTC, 0 replies.
- [PR] [CELEBORN-2064] Fix the issue where reading replica partition that returns zero chunk causes tasks to hang [celeborn] - posted by "littlexyw (via GitHub)" <gi...@apache.org> on 2025/07/11 10:37:48 UTC, 5 replies.
- Re: [PR] Set max memory of SortBasedPusher based off Spark configs [celeborn] - posted by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2025/07/13 08:36:00 UTC, 1 replies.
- [PR] [CELEBORN-2066] Release workers only with high workload when the number of excluded worker set is too large. [celeborn] - posted by "Kalvin2077 (via GitHub)" <gi...@apache.org> on 2025/07/14 09:47:53 UTC, 0 replies.
- [PR] [CELEBORN-2059] Support reusing hdfs output stream with HDFS storage [celeborn] - posted by "taowenjun (via GitHub)" <gi...@apache.org> on 2025/07/14 11:57:43 UTC, 2 replies.
- [PR] [CELEBORN-2067] Clean up deprecated Guava API usage [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/15 03:37:39 UTC, 1 replies.
- [PR] [CELEBORN-2068] TransportClientFactory should close channel explicitly to avoid resource leak for timeout or failure [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/15 07:48:37 UTC, 7 replies.
- [PR] [CELEBORN-2071] Fix the issue where some gauge metrics were not registered to the metricRegistry [celeborn] - posted by "TheodoreLx (via GitHub)" <gi...@apache.org> on 2025/07/15 12:16:39 UTC, 3 replies.
- [I] Opening an issue in Jira [celeborn] - posted by "dorian-tsorfy (via GitHub)" <gi...@apache.org> on 2025/07/15 14:15:26 UTC, 16 replies.
- Re: [PR] [CELEBORN-2049] Bump Ratis version from 3.1.3 to 3.2.0 [celeborn] - posted by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2025/07/16 08:39:11 UTC, 1 replies.
- Re: [D] Deploying Apache Celebron for Large-Scale Spark SQL Workloads [celeborn] - posted by "ssharma (via GitHub)" <gi...@apache.org> on 2025/07/16 15:09:49 UTC, 0 replies.
- [PR] buf size [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/18 01:10:30 UTC, 1 replies.
- [PR] [CELEBORN-2070][CIP-14] Support RegisterShuffle, MapperEnd/Response in CppClient [celeborn] - posted by "HolyLow (via GitHub)" <gi...@apache.org> on 2025/07/18 09:02:15 UTC, 2 replies.
- [PR] Add instance filter [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/19 03:06:48 UTC, 0 replies.
- Re: [PR] [CELEBORN-1817] add committed file size metrics [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/19 03:27:03 UTC, 1 replies.
- [PR] Fix metrics postions [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/19 03:43:02 UTC, 0 replies.
- Re: [PR] [WIP] Dynamically adjust partition write parallelism [celeborn] - posted by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2025/07/19 08:36:58 UTC, 0 replies.
- [PR] [CELEBORN-2074] Create hdfs shuffle directory by master to reduce Namenode call [celeborn] - posted by "taowenjun (via GitHub)" <gi...@apache.org> on 2025/07/19 09:05:42 UTC, 0 replies.
- [PR] Fix `OpenStreamTime` metrics for `PbOpenStreamList` request [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/20 18:38:34 UTC, 0 replies.
- Re: [PR] [CELEBORN-2075] Fix `OpenStreamTime` metrics for `PbOpenStreamList` request [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/20 19:05:09 UTC, 8 replies.
- [PR] [CELEBORN-2076] Log real chunk buf size [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/20 19:13:54 UTC, 2 replies.
- [PR] [MINOR] Fix a typo buffer to body in ChunkFetchSuccess.toString [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/21 03:10:24 UTC, 1 replies.
- Re: [PR] [CELEBORN-2073] Fix PartitionFileSizeBytes metrics [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/21 05:19:22 UTC, 2 replies.
- Re: [PR] [CELEBORN-2072] Add missing instance filter to grafana dashboard [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/21 06:27:29 UTC, 1 replies.
- Re: [PR] [CELEBORN-1917] Support celeborn.client.push.maxBytesSizeInFlight [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/21 11:18:28 UTC, 16 replies.
- Re: [PR] [CELEBORN-2044] Proactively cleanup stream state from ChunkStreamManager when the stream ends [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/21 23:20:56 UTC, 1 replies.
- Re: [PR] [CELEBORN-1640] NettyMemoryMetrics supports numHeapArenas, numDirectArenas, tinyCacheSize, smallCacheSize, normalCacheSize, numThreadLocalCaches and chunkSize [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/22 05:45:00 UTC, 0 replies.
- Re: [PR] [CELEBORN-2076] Add `FetchChunkSize` metrics [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/22 06:10:20 UTC, 2 replies.
- [PR] [CELEBORN-2077] Improve toString by JEP-280 instead of ToStringBuilder [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/22 06:14:58 UTC, 2 replies.
- [PR] Fix rpc unit [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/22 22:17:18 UTC, 0 replies.
- Re: [PR] [CELEBORN-2078] Fix wrong grafana metrics units [celeborn] - posted by "codecov[bot] (via GitHub)" <gi...@apache.org> on 2025/07/23 00:24:49 UTC, 2 replies.
- [PR] Fix netty metrics [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/23 05:30:15 UTC, 0 replies.
- Re: [PR] [CELEBORN-2079] Fix grafana dashboard netty metrics [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/23 06:28:30 UTC, 0 replies.
- [PR] Netty pool [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/23 06:52:04 UTC, 0 replies.
- [PR] [CELEBORN-2080] Bump Flink from 1.19.2, 1.20.1 to 1.19.3, 1.20.2 [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/24 03:32:57 UTC, 2 replies.
- [PR] Followup comments [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/24 04:12:59 UTC, 0 replies.
- Re: [PR] [CELEBORN-1917][FOLLOWUP] Do not wait if total inflight or inflight to a specific worker is lower than threshold [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/24 04:27:26 UTC, 4 replies.
- [PR] [CELEBORN-1793] Add netty pinned memory metrics [celeborn] - posted by "leixm (via GitHub)" <gi...@apache.org> on 2025/07/24 07:39:40 UTC, 3 replies.
- Re: [PR] [CELEBORN-2079] Fix netty grafana dashboard [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/24 07:42:18 UTC, 4 replies.
- [PR] [CELEBORN-1776] Add UT when replica enable and workers randomly shutdown [celeborn] - posted by "zaynt4606 (via GitHub)" <gi...@apache.org> on 2025/07/24 07:44:38 UTC, 1 replies.
- Re: [PR] [CELEBORN-2070][CIP-14] Support MapperEnd/Response in CppClient [celeborn] - posted by "HolyLow (via GitHub)" <gi...@apache.org> on 2025/07/24 09:28:39 UTC, 2 replies.
- [PR] [CELEBORN-2063] Parallelize the create partition writer in handleReserveSlots to speed up the reserveSlots RPC process time [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/24 12:06:45 UTC, 10 replies.
- [PR] [CELEBORN-1787] Fix netty memory metrics was not registered when enable memory allocator share [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/25 06:44:22 UTC, 12 replies.
- [PR] check unit [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/25 07:56:24 UTC, 1 replies.
- [PR] [CELEBORN-2081] PushDataHandler onFailure log shuffle key [celeborn] - posted by "cxzl25 (via GitHub)" <gi...@apache.org> on 2025/07/25 09:40:07 UTC, 2 replies.
- [PR] [CELEBORN-2082] Add the log of excluded workers with high workloads [celeborn] - posted by "cxzl25 (via GitHub)" <gi...@apache.org> on 2025/07/25 11:34:21 UTC, 2 replies.
- [PR] Log error for `recordWorkerFailure` to separate with worker status changed from application heartbeat response [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/28 03:59:54 UTC, 0 replies.
- Re: [PR] [CELEBORN-2083] For `WorkerStatusTracker`, log error for `recordWorkerFailure` [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/28 05:46:05 UTC, 1 replies.
- [PR] [CELEBORN-2084] /api/v1/workers adds tagged workers to list tagged workers information [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/28 07:16:08 UTC, 0 replies.
- [PR] [CELEBORN-2085] Use a fixed buffer for flush copying to reduce GC [celeborn] - posted by "TheodoreLx (via GitHub)" <gi...@apache.org> on 2025/07/28 08:10:33 UTC, 6 replies.
- [PR] [CELEBORN-2086] S3FlushTask and OssFlushTask should close ByteArrayInputStream to avoid resource leak [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/28 09:58:43 UTC, 3 replies.
- [PR] [CELEBORN-894][CELEBORN-474][FOLLOWUP] PushState uses JavaUtils#newConcurrentHashMap to speed up ConcurrentHashMap#computeIfAbsent [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/28 12:11:41 UTC, 2 replies.
- [PR] [CELEBORN-1971] Use synchronization as opposed to Thread.sleep for InFlightRequestTracker limit [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/29 02:54:58 UTC, 3 replies.
- [PR] [MINOR] Document introduced metrics into monitoring.md [celeborn] - posted by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/07/29 05:35:28 UTC, 2 replies.
- [PR] Celeborn config [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/29 18:02:38 UTC, 0 replies.
- Re: [PR] [CELEBORN-2087] Refine the docs configuration view [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/29 18:11:25 UTC, 1 replies.
- Re: [PR] [CELEBORN-2087] Refine the docs configuration table view [celeborn] - posted by "codecov[bot] (via GitHub)" <gi...@apache.org> on 2025/07/29 20:31:18 UTC, 3 replies.
- [D] Not Seeing Speedup with Celeborn [celeborn] - posted by "lioryankovitz (via GitHub)" <gi...@apache.org> on 2025/07/30 08:18:22 UTC, 3 replies.
- [PR] Fix npe [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/30 20:51:42 UTC, 0 replies.
- Re: [PR] [CELEBORN-2088] Fix NPE if `celeborn.client.spark.fetch.cleanFailedShuffle` enabled [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/30 21:15:04 UTC, 1 replies.
- [PR] [CELEBORN-2090] Support Lz4 Decompression in CppClient [celeborn] - posted by "Jraaay (via GitHub)" <gi...@apache.org> on 2025/07/31 07:49:14 UTC, 1 replies.
- [PR] Commit failed [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/31 18:59:14 UTC, 2 replies.
- Re: [PR] [CELEBORN-2024][FOLLOWUP] Increase COMMIT_FILES_FAIL_COUNT when closing tier writer timeout [celeborn] - posted by "turboFei (via GitHub)" <gi...@apache.org> on 2025/07/31 20:27:55 UTC, 1 replies.