You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dstandish (via GitHub)" <gi...@apache.org> on 2024/04/25 14:42:51 UTC
[PR] Determine needs_expansion on client side when DB isolated [airflow]
dstandish opened a new pull request, #39257:
URL: https://github.com/apache/airflow/pull/39257
This is necessary because Operator does not serialize quite right. It doesn't (either completely or at all) include the dag object or its groups. So we can't properly determine 'needs expansion' on server side when it's in a mapped task group.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Determine needs_expansion on client side when DB isolated [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #39257:
URL: https://github.com/apache/airflow/pull/39257#issuecomment-2078611765
> It doesn't (either completely or at all) include the dag object or its groups.
And… we somehow don’t want it to do that? I feel we need to do a better job recreating the heirarchy. The code base is getting more and more difficult to maintain with all the special cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Determine needs_expansion on client side when DB isolated [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on PR #39257:
URL: https://github.com/apache/airflow/pull/39257#issuecomment-2078024043
@uranusjr you want to have a look see here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Determine needs_expansion on client side when DB isolated [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on PR #39257:
URL: https://github.com/apache/airflow/pull/39257#issuecomment-2078635029
> > It doesn't (either completely or at all) include the dag object or its groups.
>
> And… we somehow don’t want it to do that? I feel we need to do a better job recreating the heirarchy. The code base is getting more and more difficult to maintain with all the special cases.
Yeah I agree. Yeah I mean these PRs are in part like .... illustrating problems by showing what we have to do to make things work. i'm not sure what we want to / need to / ought to do with regard to serialization of task objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Determine needs_expansion on client side when DB isolated [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on PR #39257:
URL: https://github.com/apache/airflow/pull/39257#issuecomment-2078641479
but like ... also... @uranusjr, using this example... there actually are some reasonable arguments for not serializing the complete task object. for one it's more efficient. why serialize the full task (taken to include the full dag and all other tasks) and send over the network just so that "needs expansion" can be evaluated on the server, when you can just determine locally whether it needs expansion and tell the server what to do and save the net traffic and processing time? further do we not run into recursion problem serializing at task that has the full dag when the dag also has all tasks each of which has the full dag and on and on.....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Determine needs_expansion on client side when DB isolated [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish closed pull request #39257: Determine needs_expansion on client side when DB isolated
URL: https://github.com/apache/airflow/pull/39257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Determine needs_expansion on client side when DB isolated [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on PR #39257:
URL: https://github.com/apache/airflow/pull/39257#issuecomment-2111178598
closing in favor of https://github.com/apache/airflow/pull/39604
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org