You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Ping Zhang <pi...@umich.edu> on 2022/12/20 03:02:31 UTC
[Discuss]whether fail early for Xcom.set when value exceeds the BLOB limit
Hi community,
I would like to discuss if we want to fail early for Xcom.set when the
value exceeds the storage limit. Otherwise, the database (default xcom
backend) truncates the value leading to failure when Xcom reads it, (e.g.
on UI, Admin -> Xcoms will also fail).
But I am not sure whether we want to fail the whole task execution if
Xcom.set fails and would like to have your inputs.
Thanks,
Ping
Re: [Discuss]whether fail early for Xcom.set when value exceeds the BLOB limit
Posted by "Ferruzzi, Dennis" <fe...@amazon.com.INVALID>.
100% fail early sounds like the right answer here.
________________________________
From: Ping Zhang <pi...@umich.edu>
Sent: Monday, December 19, 2022 7:02 PM
To: dev@airflow.apache.org
Subject: [EXTERNAL] [Discuss]whether fail early for Xcom.set when value exceeds the BLOB limit
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hi community,
I would like to discuss if we want to fail early for Xcom.set when the value exceeds the storage limit. Otherwise, the database (default xcom backend) truncates the value leading to failure when Xcom reads it, (e.g. on UI, Admin -> Xcoms will also fail).
But I am not sure whether we want to fail the whole task execution if Xcom.set fails and would like to have your inputs.
Thanks,
Ping
Re: [Discuss]whether fail early for Xcom.set when value exceeds the BLOB limit
Posted by "Ferruzzi, Dennis" <fe...@amazon.com.INVALID>.
Gah, sorry, I missed the other replies on this one. What they said. :P
________________________________
From: Ping Zhang <pi...@umich.edu>
Sent: Tuesday, December 20, 2022 10:43 AM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL][Discuss]whether fail early for Xcom.set when value exceeds the BLOB limit
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Thanks Jarek and Ash. I will prepare a PR for it.
Thanks,
Ping
On Tue, Dec 20, 2022 at 6:28 AM Ash Berlin-Taylor <as...@apache.org>> wrote:
+1 -- though I don wonder if that is a behaviour that only MySQL exhibits?
Either way, we should throw an exception early.
-ash
On Dec 20 2022, at 8:04 am, Jarek Potiuk <ja...@potiuk.com>> wrote:
+1. I am all for failing it. I do not see an immediate issue with it - especially if we also explain how to handle it (i.e. custom XCom, limiting data, truncating data).
I assume this is only when "standard" XCom is used. I would be for it, explicit is better than implicit. The XCom limit has always been a bit "vague" - we wrote something alongside "not too big" value and you could check in the code what the limit is but we never explicitly specified that as a limitation, but it was there.
For me this looks like even more details of something that we already started discussing separately - i.e. being explicit about the "public interface" of ours and being explicit with that. If our aim is being an "extensible platform", then we have to be very clear about what our users can and cannot do with Airflow.
J.
On Tue, Dec 20, 2022 at 4:03 AM Ping Zhang <pi...@umich.edu>> wrote:
Hi community,
I would like to discuss if we want to fail early for Xcom.set when the value exceeds the storage limit. Otherwise, the database (default xcom backend) truncates the value leading to failure when Xcom reads it, (e.g. on UI, Admin -> Xcoms will also fail).
But I am not sure whether we want to fail the whole task execution if Xcom.set fails and would like to have your inputs.
Thanks,
Ping
Re: [Discuss]whether fail early for Xcom.set when value exceeds the BLOB limit
Posted by Ping Zhang <pi...@umich.edu>.
Thanks Jarek and Ash. I will prepare a PR for it.
Thanks,
Ping
On Tue, Dec 20, 2022 at 6:28 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> +1 -- though I don wonder if that is a behaviour that only MySQL exhibits?
>
> Either way, we should throw an exception early.
>
> -ash
>
> On Dec 20 2022, at 8:04 am, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> +1. I am all for failing it. I do not see an immediate issue with it -
> especially if we also explain how to handle it (i.e. custom XCom, limiting
> data, truncating data).
>
> I assume this is only when "standard" XCom is used. I would be for it,
> explicit is better than implicit. The XCom limit has always been a bit
> "vague" - we wrote something alongside "not too big" value and you could
> check in the code what the limit is but we never explicitly specified that
> as a limitation, but it was there.
>
> For me this looks like even more details of something that we already
> started discussing separately - i.e. being explicit about the "public
> interface" of ours and being explicit with that. If our aim is being an
> "extensible platform", then we have to be very clear about what our users
> can and cannot do with Airflow.
>
> J.
>
>
> On Tue, Dec 20, 2022 at 4:03 AM Ping Zhang <pi...@umich.edu> wrote:
>
> Hi community,
>
> I would like to discuss if we want to fail early for Xcom.set when the
> value exceeds the storage limit. Otherwise, the database (default xcom
> backend) truncates the value leading to failure when Xcom reads it, (e.g.
> on UI, Admin -> Xcoms will also fail).
>
> But I am not sure whether we want to fail the whole task execution if
> Xcom.set fails and would like to have your inputs.
>
> Thanks,
>
> Ping
>
>
Re: [Discuss]whether fail early for Xcom.set when value exceeds the BLOB limit
Posted by Ash Berlin-Taylor <as...@apache.org>.
+1 -- though I don wonder if that is a behaviour that only MySQL exhibits?
Either way, we should throw an exception early.
-ash
On Dec 20 2022, at 8:04 am, Jarek Potiuk <ja...@potiuk.com> wrote:
> +1. I am all for failing it. I do not see an immediate issue with it - especially if we also explain how to handle it (i.e. custom XCom, limiting data, truncating data).
>
>
> I assume this is only when "standard" XCom is used. I would be for it, explicit is better than implicit. The XCom limit has always been a bit "vague" - we wrote something alongside "not too big" value and you could check in the code what the limit is but we never explicitly specified that as a limitation, but it was there.
>
> For me this looks like even more details of something that we already started discussing separately - i.e. being explicit about the "public interface" of ours and being explicit with that. If our aim is being an "extensible platform", then we have to be very clear about what our users can and cannot do with Airflow.
>
> J.
>
>
> On Tue, Dec 20, 2022 at 4:03 AM Ping Zhang <pingzh@umich.edu (mailto:pingzh@umich.edu)> wrote:
> > Hi community,
> >
> > I would like to discuss if we want to fail early for Xcom.set when the value exceeds the storage limit. Otherwise, the database (default xcom backend) truncates the value leading to failure when Xcom reads it, (e.g. on UI, Admin -> Xcoms will also fail).
> >
> > But I am not sure whether we want to fail the whole task execution if Xcom.set fails and would like to have your inputs.
> >
> > Thanks,
> >
> > Ping
Re: [Discuss]whether fail early for Xcom.set when value exceeds the BLOB limit
Posted by Jarek Potiuk <ja...@potiuk.com>.
+1. I am all for failing it. I do not see an immediate issue with it -
especially if we also explain how to handle it (i.e. custom XCom, limiting
data, truncating data).
I assume this is only when "standard" XCom is used. I would be for it,
explicit is better than implicit. The XCom limit has always been a bit
"vague" - we wrote something alongside "not too big" value and you could
check in the code what the limit is but we never explicitly specified that
as a limitation, but it was there.
For me this looks like even more details of something that we already
started discussing separately - i.e. being explicit about the "public
interface" of ours and being explicit with that. If our aim is being an
"extensible platform", then we have to be very clear about what our users
can and cannot do with Airflow.
J.
On Tue, Dec 20, 2022 at 4:03 AM Ping Zhang <pi...@umich.edu> wrote:
> Hi community,
>
> I would like to discuss if we want to fail early for Xcom.set when the
> value exceeds the storage limit. Otherwise, the database (default xcom
> backend) truncates the value leading to failure when Xcom reads it, (e.g.
> on UI, Admin -> Xcoms will also fail).
>
> But I am not sure whether we want to fail the whole task execution if
> Xcom.set fails and would like to have your inputs.
>
> Thanks,
>
> Ping
>