You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/12/09 14:19:04 UTC
[GitHub] [dolphinscheduler] jieguangzhou commented on issue #13133: [Feature][Master] Add task caching mechanism to improve the running speed of repetitive tasks
jieguangzhou commented on issue #13133:
URL: https://github.com/apache/dolphinscheduler/issues/13133#issuecomment-1344365660
> Hi, @jieguangzhou, wonderful idea! In addition, one thing i think should take into consideration:
>
> > A well-behaved Flyte task should generate **deterministic** output given the same inputs and task functionality.
>
> How can we make sure the task is deterministic? Inference is deterministic, but training can only be relatively deterministic because of some random steps. Also, other types of tasks are not always deterministic. Should we add task-related configuration for using this cache mechanism, user can determine whether they think the task can give deterministic output or uncertainty can be ignored, WDYT?
Yes, I think we can add a new flag to let users decide whether to use the cache task
Maybe like this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org