You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/01/17 18:12:00 UTC
[jira] [Commented] (SAMZA-1523) Cleanup table entries before
shutting down the processor
[ https://issues.apache.org/jira/browse/SAMZA-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329131#comment-16329131 ]
ASF GitHub Bot commented on SAMZA-1523:
---------------------------------------
Github user asfgit closed the pull request at:
https://github.com/apache/samza/pull/379
> Cleanup table entries before shutting down the processor
> --------------------------------------------------------
>
> Key: SAMZA-1523
> URL: https://issues.apache.org/jira/browse/SAMZA-1523
> Project: Samza
> Issue Type: Bug
> Reporter: Navina Ramesh
> Assignee: Navina Ramesh
> Priority: Major
>
> We want to remove expired entries of the processors from the Azure Table when the processor is shutting down. Azure Table service uses optimistic locking by default. Hence, when the coordinator thread is cleaning up during shutdown, it is possible for the heartbeat thread to update that entry as well. This causes a failure in cleanup and throws exceptions in the log. Obviously, it also fails to clear the entries :)
> `
> 2017-11-30 15:23:32.804 [JMVersionUpgradeScheduler-0] AzureJobCoordinator [INFO] pid=05133d0a-dd85-4178-a97c-2c98dc617308new version 5 of the job model got confirmed
> 2017-11-30 15:23:32.833 [HeartbeatScheduler-0] HeartbeatScheduler [INFO] Updating heartbeat for processor ID: 05133d0a-dd85-4178-a97c-2c98dc617308 and job model version: 4
> 2017-11-30 15:23:32.905 [JMVersionUpgradeScheduler-0] TableUtils [ERROR] Azure storage exception while deleting processor entity with job model version: 4and pid: 05133d0a-dd85-4178-a97c-2c98dc617308
> com.microsoft.azure.storage.table.TableServiceException: Precondition Failed
> at com.microsoft.azure.storage.table.TableServiceException.generateTableServiceException(TableServiceException.java:52)
> at com.microsoft.azure.storage.table.TableOperation$1.preProcessResponse(TableOperation.java:319)
> at com.microsoft.azure.storage.table.TableOperation$1.preProcessResponse(TableOperation.java:299)
> at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:139)
> at com.microsoft.azure.storage.table.TableOperation.performDelete(TableOperation.java:281)
> at com.microsoft.azure.storage.table.TableOperation.execute(TableOperation.java:685)
> at com.microsoft.azure.storage.table.CloudTable.execute(CloudTable.java:529)
> at com.microsoft.azure.storage.table.CloudTable.execute(CloudTable.java:496)
> at org.apache.samza.util.TableUtils.deleteProcessorEntity(TableUtils.java:157)
> at org.apache.samza.coordinator.AzureJobCoordinator.onNewJobModelConfirmed(AzureJobCoordinator.java:448)
> at org.apache.samza.coordinator.AzureJobCoordinator.onNewJobModelAvailable(AzureJobCoordinator.java:419)
> at org.apache.samza.coordinator.AzureJobCoordinator.lambda$createJMVersionUpgradeListener$3(AzureJobCoordinator.java:248)
> at org.apache.samza.coordinator.scheduler.JMVersionUpgradeScheduler.lambda$scheduleTask$0(JMVersionUpgradeScheduler.java:81)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-11-30 15:23:32.906 [JMVersionUpgradeScheduler-0] AzureJobCoordinator [ERROR] Exception in Job Model Version Upgrade Scheduler. Stopping the processor...
> `
> We should disable optimistic locking during the cleanup phase of shutdown. Ideal solution is to perhaps have more control over the various schedulers. That is beyond the scope of this JIRA though :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)