You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ajeydudhe (via GitHub)" <gi...@apache.org> on 2024/09/24 08:18:15 UTC

[I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

ajeydudhe opened a new issue, #14067:
URL: https://github.com/apache/pinot/issues/14067

   ### Steps to reproduce
   
   - Define REALTIME table schema and table config using Pinot **_1.2.0_**
   - In the table config, make sure the **_upsertConfig_** has **_enableSnapshot = true_** and **_enablePreload = true_** as below:
   ```json 
   "upsertConfig": {
           "mode": "FULL",
           "comparisonColumns": ["someField"],
           "partialUpsertStrategies": {},
           "deleteRecordColumn": "isDeleted",
           "deletedKeysTTL": 86400000,
           "enableSnapshot": true,
           "enablePreload": true
   }
   ```
   - Load some records using Kafka and check that data is visible and all segments are GOOD.
   - Now, pause the Kafka consumption using following command:
   ```shell
   curl --location --request POST 'http://localhost:9000/tables/myTable/pauseConsumption'
   ```
   - Make sure consumption is paused using following command:
   ```shell
   curl --location 'http://localhost:9000/tables/myTable/pauseStatus'
   ```
   - Resume the consumption using following command:
   ```shell
   curl --location --request POST 'http://localhost:9000/tables/myTable/resumeConsumption'
   ```
   - Now, check the segment status.
   - All the older segments are marked BAD and new consuming ones as GOOD.
   - The server logs has following error for segments failing to load:
   ```
   2024-09-24 13:21:19 Caused by: java.lang.IllegalStateException: Snapshot of validDocIds is required to preload segment: myTable__0__0__20240924T0750Z, table: myTable_REALTIME
   ```
   
   ### Note
   
   - Same POC runs properly without any errors on Pinot **_1.1.0_**
   - Issue occurs only when using **_upsertConfig.enableSnapshot = true_** and **_upsertConfig.enablePreload = true_**
   
   ### Exception
   ```
   2024-09-24 13:21:19 2024/09/24 07:51:19.021 ERROR [HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_thread_10] Exception while executing a state transition task myTable__0__0__20240924T0750Z
   2024-09-24 13:21:19 java.lang.reflect.InvocationTargetException: null
   2024-09-24 13:21:19     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
   2024-09-24 13:21:19     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
   2024-09-24 13:21:19     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
   2024-09-24 13:21:19     at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
   2024-09-24 13:21:19     at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
   2024-09-24 13:21:19     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
   2024-09-24 13:21:19     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
   2024-09-24 13:21:19     at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
   2024-09-24 13:21:19 Caused by: java.lang.IllegalStateException: Snapshot of validDocIds is required to preload segment: myTable__0__0__20240924T0750Z, table: myTable_REALTIME
   2024-09-24 13:21:19     at org.apache.pinot.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:838) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at org.apache.pinot.segment.local.upsert.BasePartitionUpsertMetadataManager.doPreloadSegment(BasePartitionUpsertMetadataManager.java:513) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at org.apache.pinot.segment.local.upsert.BasePartitionUpsertMetadataManager.preloadSegment(BasePartitionUpsertMetadataManager.java:498) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.handleUpsert(RealtimeTableDataManager.java:585) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:541) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.downloadAndReplaceConsumingSegment(RealtimeTableDataManager.java:628) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.downloadSegmentAndReplace(RealtimeSegmentDataManager.java:1351) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.goOnlineFromConsuming(RealtimeSegmentDataManager.java:1290) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.doAddOnlineSegment(RealtimeTableDataManager.java:421) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at org.apache.pinot.core.data.manager.BaseTableDataManager.addOnlineSegment(BaseTableDataManager.java:313) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOnlineSegment(HelixInstanceDataManager.java:275) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromConsuming(SegmentOnlineOfflineStateModelFactory.java:88) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
   2024-09-24 13:21:19     ... 12 more
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2372488169

   hi @dang-stripe @ajeydudhe 
   
   could you try to set this server config `max.segment.preload.threads` a positive value (like 8 or 2x the cpu cores). When set enablePreload to true, this preloading thread pool should be set too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "dang-stripe (via GitHub)" <gi...@apache.org>.
dang-stripe commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2372473553

   We've managed to hit this exception in our test clusters on 1.2 as well w/ snapshot + preload. I haven't been able to reproduce it yet though. The segments were in ERROR state.
   
   The only relevant logs seem to be these snapshot failures, but I do not see any `Caught exception while taking snapshot` logs from https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java
   ```
   [2024-09-23 15:15:23.682414] INFO [setup_attempts_REALTIME-327710-ConcurrentMapPartitionUpsertMetadataManager] [setup_attempts__327710__147__20240923T1515Z:138928] Taking snapshot for 147 segments
   [2024-09-23 15:15:23.682430] INFO [ImmutableSegmentImpl] [setup_attempts__327710__147__20240923T1515Z:25] Persisted validDocIds for segment: setup_attempts__327710__146__20240923T1115Z with: 3 valid docs
   [2024-09-23 15:15:23.682447] WARN [setup_attempts_REALTIME-327710-ConcurrentMapPartitionUpsertMetadataManager] [setup_attempts__327710__147__20240923T1515Z:138928] Missed taking snapshot for 146 immutable segments
   [2024-09-23 15:15:23.682466] INFO [setup_attempts_REALTIME-327710-ConcurrentMapPartitionUpsertMetadataManager] [setup_attempts__327710__147__20240923T1515Z:138928] Finished taking snapshot for 1 immutable segments with 3 primary keys (out of 147 total segments, 0 are consuming segments) in 1 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2375333465

   btw, I updated pinot docs about preloading to watch out this bug in v1.2.0: https://docs.pinot.apache.org/basics/data-import/upsert#enable-preload-for-faster-server-restarts 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2372388917

   @klsince @tibrewalpratik17 Can you help take a look? We probably need to loose the check


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "ajeydudhe (via GitHub)" <gi...@apache.org>.
ajeydudhe commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2372872901

   > @klsince any idea why is this specifically occuring in 1.2 and not 1.1. It should have ideally happened in 1.1 as well as this check has been there since beginning: https://github.com/apache/pinot/pull/11020/files#diff-0a831c7605c3a8b7058200545bd732eb614526a4b03e4a972af66058874b507bR206-R207
   > 
   > > All the older segments are marked BAD and new consuming ones as GOOD.
   > 
   > @ajeydudhe can you also share if the segments are in ERROR state or marked as OFFLINE in External view?
   
   All the segments are in ERROR state.
   ### Additional info
   - I am using the attached docker compose for Pinot using following command:
   ```shell
   docker-compose up --scale server=2 -d
   ```
   - For Kafka, I am creating topic with 10 partitions and hence the table creates 10 consuming segments. But same issue is observed with 2 partitions.
   
   [Pinot Docker.zip](https://github.com/user-attachments/files/17124740/Pinot.Docker.zip)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2372503392

   the default 0 should have disabled the preloading, but in 1.2.0 or earlier, it's not disabled cleanly, because an internal flag _isPreloading flag didn't consider the pool size. The fix https://github.com/apache/pinot/pull/13747 should have corrected this, so that if pool size is 0, the preloading mechanism is disabled cleanly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "ajeydudhe (via GitHub)" <gi...@apache.org>.
ajeydudhe commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2373362425

   > @ajeydudhe could you try to set this server config max.segment.preload.threads to a positive value like 2x the cpu cores.
   > 
   > @tibrewalpratik17 I think this started to happen in 1.2.0 might be due to the refactoring of the preloading logic between 1.1.0 and 1.2.0, and the way to init the _isPreloading boolean flag was changed. Should have a test on this edge condition.
   
   Thanks @klsince. The issue is not seen if the max.segment.preload.threads config is set. I tested with max.segment.preload.threads = 2. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "dang-stripe (via GitHub)" <gi...@apache.org>.
dang-stripe commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2372595998

   thanks we'll pull in that fix. i was able to repro the issue by rebalancing the table so segments moved to servers that didn't have `max.segment.preload.threads` set and those segments came up in ERROR state.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince closed issue #14067: [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment"
URL: https://github.com/apache/pinot/issues/14067


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2377466606

   @dang-stripe yup, that should be the expected behavior today. [Here is the key try-catch block](https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java#L250) so that if preloading failed in the middle, the remaining segments would get loaded with the normal (though more costly) upsert flow (doing map get/check/put). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2377552041

   I'm closing this issue for now. And we can continue the discussions about preloading in general in the Pinot community Slack (my slack id: `Xiaobing`)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2372420538

   @klsince any idea why is this specifically occuring in 1.2 and not 1.1. It should have ideally happened in 1.1 as well as this check has been there since beginning: https://github.com/apache/pinot/pull/11020/files#diff-0a831c7605c3a8b7058200545bd732eb614526a4b03e4a972af66058874b507bR206-R207
   
   
   > All the older segments are marked BAD and new consuming ones as GOOD.
   
   @ajeydudhe can you also share if the segments are in ERROR state or marked as OFFLINE in External view? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2372439699

   I'm wondering why resuming the consumption caused old segments to be added again. It seems like adding old segments had entered the preloading logic flow unexpectedly.
   
   I'll see if I can repro this on my local. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2372482803

   `HelixTaskExecutor-message_handle_thread_10` thread was preloading the segment, and this is already unexpected. 
   
   segment preloading should be done by a separate preloading thread pool whose thread name should be like 'segment-preload-thread-xx'
   
   so somehow server entered the preloading logic unexpected, and did it on helix thread unexpectedly too.
   
   this reminded me of a fix I did weeks ago (but after 1.2.0): https://github.com/apache/pinot/pull/13747 to refine the check of enabling segment preloading. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2375316590

   add a test case to keep checking on this: https://github.com/apache/pinot/pull/14085, although the issue was fixed by that earlier PR. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "dang-stripe (via GitHub)" <gi...@apache.org>.
dang-stripe commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2375773920

   thanks @klsince - this is separate from this issue, but is it possible to make preloading fail more gracefully? ex. if any exception is thrown during preloading, fallback to non-preloading behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2372896142

   @ajeydudhe could you try to set this server config max.segment.preload.threads to a positive value like 2x the cpu cores. 
   
   @tibrewalpratik17 I think this started to happen in 1.2.0 might be due to the refactoring of the preloading logic between 1.1.0 and 1.2.0, and the way to init the _isPreloading boolean flag was changed. Should have a test on this edge condition.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] [Pinot 1.2.0] Realtime table segment reload fails with error "Snapshot of validDocIds is required to preload segment" [pinot]

Posted by "dang-stripe (via GitHub)" <gi...@apache.org>.
dang-stripe commented on issue #14067:
URL: https://github.com/apache/pinot/issues/14067#issuecomment-2372497668

   actually i wouldn't be surprised if that's the issue. we had that turned on for one of our servers and not the others. we clearly see the error on a server that didn't have this on, while preload was enabled on the table. we've since rolled out the threads setting to all the servers and haven't been able to repro the behavior.
   
   let me try to re-test w/ it off. is it possible to set a sane default for preload threads in code or make the server fail fast instead?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org