You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "perry2of5 (via GitHub)" <gi...@apache.org> on 2024/08/19 21:42:40 UTC
[PR] Add callback to process Azure Service Bus message contents [airflow]
perry2of5 opened a new pull request, #41601:
URL: https://github.com/apache/airflow/pull/41601
This PR adds a callback to process messages from an Azure Service Bus.
Right now, on main, the Azure Service Bus message receiver simply logs the messages received and returns. This PR preserves that default, but adds the ability to pass in a callback function with the signature ServiceBusMessage -> None which is applied to each message received.
This change is made to both the general receiver and the subscription receiver.
I'd like someone from the Airflow community to weigh in on if this is a good solution. To me, it seemed like the least intrusive way to add a hook to obtain the message contents. However, I'm not sure if it plays well with the Airflow Operator standards as I've barely been using Airflow for a month.
Please let me know if you see any errors, omissions, or if I'm headed in the wrong direction.
Thank you,
Tim
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on PR #41601:
URL: https://github.com/apache/airflow/pull/41601#issuecomment-2329917131
This is ready to be reviewed again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on PR #41601:
URL: https://github.com/apache/airflow/pull/41601#issuecomment-2299839998
This addresses https://github.com/apache/airflow/discussions/26446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on code in PR #41601:
URL: https://github.com/apache/airflow/pull/41601#discussion_r1744584448
##########
airflow/providers/microsoft/azure/hooks/asb.py:
##########
@@ -326,5 +342,32 @@ def receive_subscription_message(
max_message_count=max_message_count, max_wait_time=max_wait_time
)
for msg in received_msgs:
- self.log.info(msg)
- subscription_receiver.complete_message(msg)
+ self._process_message(msg, message_callback, subscription_receiver)
+
+ def _process_message(
+ self,
+ msg: ServiceBusReceivedMessage,
+ message_callback: MessageCallback | None,
+ receiver: ServiceBusReceiver,
+ ):
+ """
+ Process the message by calling the message_callback or logging the message.
+
+ :param msg: The message to process.
+ :param message_callback: Optional callback to process each message. If not provided, then
+ the message will be logged and completed. If provided, and throws an exception, the
+ message will be abandoned for future redelivery.
+ :param receiver: The receiver that received the message.
+ """
+ if message_callback is None:
+ self.log.info(msg)
Review Comment:
That is the original code https://github.com/apache/airflow/blob/main/airflow/providers/microsoft/azure/hooks/asb.py#L292
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on PR #41601:
URL: https://github.com/apache/airflow/pull/41601#issuecomment-2330269872
Thank you @potiuk and @Lee-W.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #41601:
URL: https://github.com/apache/airflow/pull/41601#discussion_r1738376359
##########
airflow/providers/microsoft/azure/hooks/asb.py:
##########
@@ -326,5 +336,27 @@ def receive_subscription_message(
max_message_count=max_message_count, max_wait_time=max_wait_time
)
for msg in received_msgs:
- self.log.info(msg)
- subscription_receiver.complete_message(msg)
+ self._process_message(msg, message_callback, subscription_receiver)
+
+ def _process_message(self, msg, message_callback, receiver):
Review Comment:
Could we add type annotation here? Thanks!
##########
airflow/providers/microsoft/azure/hooks/asb.py:
##########
@@ -270,14 +273,21 @@ def send_batch_message(sender: ServiceBusSender, messages: list[str]):
sender.send_messages(batch_message)
def receive_message(
- self, queue_name, max_message_count: int | None = 1, max_wait_time: float | None = None
+ self,
+ queue_name,
Review Comment:
I guess this should be a str?
```suggestion
queue_name: str,
```
##########
airflow/providers/microsoft/azure/hooks/asb.py:
##########
@@ -326,5 +336,27 @@ def receive_subscription_message(
max_message_count=max_message_count, max_wait_time=max_wait_time
)
for msg in received_msgs:
- self.log.info(msg)
- subscription_receiver.complete_message(msg)
+ self._process_message(msg, message_callback, subscription_receiver)
+
+ def _process_message(self, msg, message_callback, receiver):
+ """
+ Process the message by calling the message_callback or logging the message.
+
+ :param msg: The message to process.
+ :param message_callback: Optional callback to process each message. If not provided, then
+ the message will be logged and completed. If provided, and throws an exception, the
+ message will be abandoned for future redelivery.
+ :param receiver: The receiver that received the message.
+ """
+ print("message_callback:", message_callback)
+ if message_callback is None:
+ self.log.info(msg)
+ receiver.complete_message(msg)
+ else:
+ try:
+ message_callback(msg)
+ receiver.complete_message(msg)
+ except Exception as e:
+ self.log.error("Error processing message: %s", e)
+ receiver.abandon_message(msg)
+ raise e
Review Comment:
I guess we don't expect `message_callback` to raise an exception here
```suggestion
try:
message_callback(msg)
except Exception as e:
self.log.error("Error processing message: %s", e)
receiver.abandon_message(msg)
raise e
else:
receiver.complete_message(msg)
```
##########
airflow/providers/microsoft/azure/hooks/asb.py:
##########
@@ -326,5 +336,27 @@ def receive_subscription_message(
max_message_count=max_message_count, max_wait_time=max_wait_time
)
for msg in received_msgs:
- self.log.info(msg)
- subscription_receiver.complete_message(msg)
+ self._process_message(msg, message_callback, subscription_receiver)
+
+ def _process_message(self, msg, message_callback, receiver):
+ """
+ Process the message by calling the message_callback or logging the message.
+
+ :param msg: The message to process.
+ :param message_callback: Optional callback to process each message. If not provided, then
+ the message will be logged and completed. If provided, and throws an exception, the
+ message will be abandoned for future redelivery.
+ :param receiver: The receiver that received the message.
+ """
+ print("message_callback:", message_callback)
Review Comment:
We probably would like to use logging instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #41601:
URL: https://github.com/apache/airflow/pull/41601
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on PR #41601:
URL: https://github.com/apache/airflow/pull/41601#issuecomment-2436114073
opened 43361
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on code in PR #41601:
URL: https://github.com/apache/airflow/pull/41601#discussion_r1744584448
##########
airflow/providers/microsoft/azure/hooks/asb.py:
##########
@@ -326,5 +342,32 @@ def receive_subscription_message(
max_message_count=max_message_count, max_wait_time=max_wait_time
)
for msg in received_msgs:
- self.log.info(msg)
- subscription_receiver.complete_message(msg)
+ self._process_message(msg, message_callback, subscription_receiver)
+
+ def _process_message(
+ self,
+ msg: ServiceBusReceivedMessage,
+ message_callback: MessageCallback | None,
+ receiver: ServiceBusReceiver,
+ ):
+ """
+ Process the message by calling the message_callback or logging the message.
+
+ :param msg: The message to process.
+ :param message_callback: Optional callback to process each message. If not provided, then
+ the message will be logged and completed. If provided, and throws an exception, the
+ message will be abandoned for future redelivery.
+ :param receiver: The receiver that received the message.
+ """
+ if message_callback is None:
+ self.log.info(msg)
Review Comment:
That is the original code https://github.com/apache/airflow/blob/main/airflow/providers/microsoft/azure/hooks/asb.py#L292
I kept the original functionality intact but added the ability to pass a callback instead. Not really sure why printing the message to the log was the original functionality, but it seems a little silly to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on code in PR #41601:
URL: https://github.com/apache/airflow/pull/41601#discussion_r1738893954
##########
airflow/providers/microsoft/azure/hooks/asb.py:
##########
@@ -326,5 +336,27 @@ def receive_subscription_message(
max_message_count=max_message_count, max_wait_time=max_wait_time
)
for msg in received_msgs:
- self.log.info(msg)
- subscription_receiver.complete_message(msg)
+ self._process_message(msg, message_callback, subscription_receiver)
+
+ def _process_message(self, msg, message_callback, receiver):
+ """
+ Process the message by calling the message_callback or logging the message.
+
+ :param msg: The message to process.
+ :param message_callback: Optional callback to process each message. If not provided, then
+ the message will be logged and completed. If provided, and throws an exception, the
+ message will be abandoned for future redelivery.
+ :param receiver: The receiver that received the message.
+ """
+ print("message_callback:", message_callback)
Review Comment:
I'll delete that. I used print as a flag that it was temporary. Sorry about that....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #41601:
URL: https://github.com/apache/airflow/pull/41601#discussion_r1744565440
##########
airflow/providers/microsoft/azure/hooks/asb.py:
##########
@@ -326,5 +342,32 @@ def receive_subscription_message(
max_message_count=max_message_count, max_wait_time=max_wait_time
)
for msg in received_msgs:
- self.log.info(msg)
- subscription_receiver.complete_message(msg)
+ self._process_message(msg, message_callback, subscription_receiver)
+
+ def _process_message(
+ self,
+ msg: ServiceBusReceivedMessage,
+ message_callback: MessageCallback | None,
+ receiver: ServiceBusReceiver,
+ ):
+ """
+ Process the message by calling the message_callback or logging the message.
+
+ :param msg: The message to process.
+ :param message_callback: Optional callback to process each message. If not provided, then
+ the message will be logged and completed. If provided, and throws an exception, the
+ message will be abandoned for future redelivery.
+ :param receiver: The receiver that received the message.
+ """
+ if message_callback is None:
+ self.log.info(msg)
Review Comment:
Is this right thing to print info for each callback? That looks pretty eccessive.
##########
airflow/providers/microsoft/azure/hooks/asb.py:
##########
@@ -326,5 +342,32 @@ def receive_subscription_message(
max_message_count=max_message_count, max_wait_time=max_wait_time
)
for msg in received_msgs:
- self.log.info(msg)
- subscription_receiver.complete_message(msg)
+ self._process_message(msg, message_callback, subscription_receiver)
+
+ def _process_message(
+ self,
+ msg: ServiceBusReceivedMessage,
+ message_callback: MessageCallback | None,
+ receiver: ServiceBusReceiver,
+ ):
+ """
+ Process the message by calling the message_callback or logging the message.
+
+ :param msg: The message to process.
+ :param message_callback: Optional callback to process each message. If not provided, then
+ the message will be logged and completed. If provided, and throws an exception, the
+ message will be abandoned for future redelivery.
+ :param receiver: The receiver that received the message.
+ """
+ if message_callback is None:
+ self.log.info(msg)
Review Comment:
Is this right thing to print info for each callback? That looks pretty excessive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #41601:
URL: https://github.com/apache/airflow/pull/41601#discussion_r1744585720
##########
airflow/providers/microsoft/azure/hooks/asb.py:
##########
@@ -326,5 +342,32 @@ def receive_subscription_message(
max_message_count=max_message_count, max_wait_time=max_wait_time
)
for msg in received_msgs:
- self.log.info(msg)
- subscription_receiver.complete_message(msg)
+ self._process_message(msg, message_callback, subscription_receiver)
+
+ def _process_message(
+ self,
+ msg: ServiceBusReceivedMessage,
+ message_callback: MessageCallback | None,
+ receiver: ServiceBusReceiver,
+ ):
+ """
+ Process the message by calling the message_callback or logging the message.
+
+ :param msg: The message to process.
+ :param message_callback: Optional callback to process each message. If not provided, then
+ the message will be logged and completed. If provided, and throws an exception, the
+ message will be abandoned for future redelivery.
+ :param receiver: The receiver that received the message.
+ """
+ if message_callback is None:
+ self.log.info(msg)
Review Comment:
I see. Fine then :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #41601:
URL: https://github.com/apache/airflow/pull/41601#issuecomment-2314777532
It looks good at the first glance but would like to know what would be the main usage of this functionality
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on code in PR #41601:
URL: https://github.com/apache/airflow/pull/41601#discussion_r1738975370
##########
airflow/providers/microsoft/azure/hooks/asb.py:
##########
@@ -326,5 +336,27 @@ def receive_subscription_message(
max_message_count=max_message_count, max_wait_time=max_wait_time
)
for msg in received_msgs:
- self.log.info(msg)
- subscription_receiver.complete_message(msg)
+ self._process_message(msg, message_callback, subscription_receiver)
+
+ def _process_message(self, msg, message_callback, receiver):
Review Comment:
yes, added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on PR #41601:
URL: https://github.com/apache/airflow/pull/41601#issuecomment-2414315471
Everything except releasing a major version sounds too hacky to me as well.... I'll open a ticket for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on PR #41601:
URL: https://github.com/apache/airflow/pull/41601#issuecomment-2307866016
Any chance @Lee-W or @Taragolis can review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on PR #41601:
URL: https://github.com/apache/airflow/pull/41601#issuecomment-2315811790
For my particular use case, I want to grab a file-location URL out of the message body and pass it to the next task in the DAG. In general, it provides a way to receive data from the message and react to it. This data may be success or failure information, file locations, etc. Other people have looked for this functionality too: https://github.com/apache/airflow/discussions/26446
In the AWS provider, the SQS sensor provides a similar result by putting the message into XCOM under the key messages. https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/sensors/sqs.py#L46 I don't know that I always want the entire message shoved into XCOM though. Often, I might want to branch or just store part of the data and so I provide a callback function so the user of the operator can choose what they want to happen with the data in the message body.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on PR #41601:
URL: https://github.com/apache/airflow/pull/41601#issuecomment-2411644710
I might have messed this one up from a usability point of view. I was assuming that if the context was needed in the callback it could be captured in a lambda and passed down. However, I'm having trouble accomplishing that and it look like the old technique of calling an operator from within an "outer" task function is now _highly_ discouraged.
The obvious solution"is to add a context parameter to the callback function. However, this would change the method signature and potentially break code somewhere "in the wild". I highly doubt anyone is using this yet, but that seems like the wrong thing to do.
Another solution would be to have to possible signatures of the callback: one with the context parameter and one without. Then inspect the callback function and only pass the context if the callback takes two arguments. That seems messy. Any thoughts on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on PR #41601:
URL: https://github.com/apache/airflow/pull/41601#issuecomment-2411751647
I suppose for 1-argument callbacks we could catch the type error, print a warning to update the function, and then call it with just the message. Is generating the TypeError expensive?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #41601:
URL: https://github.com/apache/airflow/pull/41601#issuecomment-2412632998
> The obvious solution"is to add a context parameter to the callback function. However, this would change the method signature and potentially break code somewhere "in the wild". I highly doubt anyone is using this yet, but that seems like the wrong thing to do.
maybe bump a major version for this?
> Another solution would be to have to possible signatures of the callback: one with the context parameter and one without. Then inspect the callback function and only pass the context if the callback takes two arguments. That seems messy. Any thoughts on this?
I personally don't like it 🤔 sounds hacky to me
> I suppose for 1-argument callbacks we could catch the type error, print a warning to update the function, and then call it with just the message. Is generating the TypeError expensive?
I guess it just as expensive as a regular exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on code in PR #41601:
URL: https://github.com/apache/airflow/pull/41601#discussion_r1738925961
##########
airflow/providers/microsoft/azure/hooks/asb.py:
##########
@@ -326,5 +336,27 @@ def receive_subscription_message(
max_message_count=max_message_count, max_wait_time=max_wait_time
)
for msg in received_msgs:
- self.log.info(msg)
- subscription_receiver.complete_message(msg)
+ self._process_message(msg, message_callback, subscription_receiver)
+
+ def _process_message(self, msg, message_callback, receiver):
+ """
+ Process the message by calling the message_callback or logging the message.
+
+ :param msg: The message to process.
+ :param message_callback: Optional callback to process each message. If not provided, then
+ the message will be logged and completed. If provided, and throws an exception, the
+ message will be abandoned for future redelivery.
+ :param receiver: The receiver that received the message.
+ """
+ print("message_callback:", message_callback)
+ if message_callback is None:
+ self.log.info(msg)
+ receiver.complete_message(msg)
+ else:
+ try:
+ message_callback(msg)
+ receiver.complete_message(msg)
+ except Exception as e:
+ self.log.error("Error processing message: %s", e)
+ receiver.abandon_message(msg)
+ raise e
Review Comment:
I made this change.
Could you describe the advantages to putting the call to "complete_message" in the "else:" rather than at the end of the try? I was thinking of completing the message failed we'd want to log an error and abandon it. Possibly that is going to fail too though if completing it failed. Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add callback to process Azure Service Bus message contents [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #41601:
URL: https://github.com/apache/airflow/pull/41601#issuecomment-2320763837
> For my particular use case, I want to grab a file-location URL out of the message body and pass it to the next task in the DAG. In general, it provides a way to receive data from the message and react to it. This data may be success or failure information, file locations, etc. Other people have looked for this functionality too: #26446
>
> In the AWS provider, the SQS sensor provides a similar result by putting the message into XCOM under the key messages. https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/sensors/sqs.py#L46 I don't know that I always want the entire message shoved into XCOM though. Often, I might want to branch or just store part of the data and so I provide a callback function so the user of the operator can choose what they want to happen with the data in the message body.
Sounds good 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org