You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "AchimGaedkeLynker (via GitHub)" <gi...@apache.org> on 2024/01/17 00:46:18 UTC
[PR] EC2 CreateInstance: terminate instances in on_kill [airflow]
AchimGaedkeLynker opened a new pull request, #36828:
URL: https://github.com/apache/airflow/pull/36828
This PR implements the termination of instances in creation when a task instance is stopped (externally marked for retry) before the instance IDs were returned. It uses the `on_kill` method to shut down EC2 instances.
Use case/problem:
The `EC2CreateInstanceOperator` is often used in a setup task of https://airflow.apache.org/docs/apache-airflow/stable/howto/setup-and-teardown.html . If the task inside the DAG is cancelled, a (potentially) ongoing `EC2CreateInstanceOperator` task is killed. This is likely if the AWS initialization and/or the post-processing step take considerable time (large instance storage, long cloud-init processes) and the `wait_for_completion` is `True`.
Without the `on_kill` cleanup code the partially initialized instances (i.e. the instance ids were not sent to XCom) will not be terminated by the tear-down task.
This happened several times to me and I finally found the cause and fixed it.
Alternative: Split the setup into many small tasks: `EC2CreateInstance` as a setup (wo `wait_for_completion`), wait on instance state, wait on cloud-init setup to finish, ... `EC2TerminateInstance` as teardown.
Questions:
* Is it required to delete the `_on_kill_instance_ids` - what's coming after execute in the TI lifecycle?
* How to unit-test this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] EC2 CreateInstance: terminate instances in on_kill [airflow]
Posted by "syedahsn (via GitHub)" <gi...@apache.org>.
syedahsn commented on code in PR #36828:
URL: https://github.com/apache/airflow/pull/36828#discussion_r1456470327
##########
airflow/providers/amazon/aws/operators/ec2.py:
##########
@@ -183,22 +183,36 @@ def execute(self, context: Context):
MaxCount=self.max_count,
**self.config,
)["Instances"]
- instance_ids = []
- for instance in instances:
- instance_ids.append(instance["InstanceId"])
- self.log.info("Created EC2 instance %s", instance["InstanceId"])
+
+ instance_ids = self._on_kill_instance_ids = [instance["InstanceId"] for instance in instances]
Review Comment:
I'm not sure if I'm understanding this correctly, but what is the purpose of `_on_kill_instance_ids`? `instance_ids` is not modified anywhere (that I can see), so wouldn't they both be the same. Would it be better to just define `self.instance_ids`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] EC2 CreateInstance: terminate instances in on_kill [airflow]
Posted by "syedahsn (via GitHub)" <gi...@apache.org>.
syedahsn commented on code in PR #36828:
URL: https://github.com/apache/airflow/pull/36828#discussion_r1457843632
##########
airflow/providers/amazon/aws/operators/ec2.py:
##########
@@ -183,22 +183,36 @@ def execute(self, context: Context):
MaxCount=self.max_count,
**self.config,
)["Instances"]
- instance_ids = []
- for instance in instances:
- instance_ids.append(instance["InstanceId"])
- self.log.info("Created EC2 instance %s", instance["InstanceId"])
+
+ instance_ids = self._on_kill_instance_ids = [instance["InstanceId"] for instance in instances]
Review Comment:
Hmm, I honestly didn't consider that perspective, but I think your approach is correct - better to be cautious and have a potentially redundant variable rather than introducing a race condition of some sort. Thanks for explaining your thoughts, I appreciate it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] EC2 CreateInstance: terminate instances in on_kill [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck merged PR #36828:
URL: https://github.com/apache/airflow/pull/36828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] EC2 CreateInstance: terminate instances in on_kill [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #36828:
URL: https://github.com/apache/airflow/pull/36828#issuecomment-1896244392
> > Can you add the test cases for this?
>
> Absolutely, I just have to find out how to test `on_kill`... It's on my todo list. Will dig into the AWS mock setup...
You can call the `on_kill` function as any other function ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] EC2 CreateInstance: terminate instances in on_kill [airflow]
Posted by "AchimGaedkeLynker (via GitHub)" <gi...@apache.org>.
AchimGaedkeLynker commented on code in PR #36828:
URL: https://github.com/apache/airflow/pull/36828#discussion_r1456505130
##########
airflow/providers/amazon/aws/operators/ec2.py:
##########
@@ -183,22 +183,36 @@ def execute(self, context: Context):
MaxCount=self.max_count,
**self.config,
)["Instances"]
- instance_ids = []
- for instance in instances:
- instance_ids.append(instance["InstanceId"])
- self.log.info("Created EC2 instance %s", instance["InstanceId"])
+
+ instance_ids = self._on_kill_instance_ids = [instance["InstanceId"] for instance in instances]
Review Comment:
This type of clean-up in case of an error is very similar to the purpose of [context/with statements in python](https://realpython.com/python-with-statement/#the-with-statement-approach)
`on_kill` is called outside the normal execution sequence as part of the signal handling for `SIGTERM` - hence the extra variable only for that purpose.
Yes, one could call it `self.instance_ids` - I actually considered it - this depends on the lifecycle of the `TaskInstance` object, call order of the methods. So the list of instances to clean up should be cleared when `XCom` has communicated the result, I assumed that happens last. I named the variable very purpose specific.
Would you think it would be necessary to delete the `_on_kill_instance_ids`, e.g. as part of the `post_execute` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] EC2 CreateInstance: terminate instances in on_kill [airflow]
Posted by "AchimGaedkeLynker (via GitHub)" <gi...@apache.org>.
AchimGaedkeLynker commented on PR #36828:
URL: https://github.com/apache/airflow/pull/36828#issuecomment-1894882544
> Can you add the test cases for this?
Absolutely, I just have to find out how to test `on_kill`... It's on my todo list, any pointers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] EC2 CreateInstance: terminate instances in on_kill [airflow]
Posted by "AchimGaedkeLynker (via GitHub)" <gi...@apache.org>.
AchimGaedkeLynker commented on code in PR #36828:
URL: https://github.com/apache/airflow/pull/36828#discussion_r1456505130
##########
airflow/providers/amazon/aws/operators/ec2.py:
##########
@@ -183,22 +183,36 @@ def execute(self, context: Context):
MaxCount=self.max_count,
**self.config,
)["Instances"]
- instance_ids = []
- for instance in instances:
- instance_ids.append(instance["InstanceId"])
- self.log.info("Created EC2 instance %s", instance["InstanceId"])
+
+ instance_ids = self._on_kill_instance_ids = [instance["InstanceId"] for instance in instances]
Review Comment:
This type of clean-up in case of an error is very similar to the purpose of [context/with statements in python](https://realpython.com/python-with-statement/#the-with-statement-approach)
`on_kill` is called outside the normal execution sequence as part of the signal handling for `SIGTERM` - hence the extra variable only for that purpose.
Yes, one could call it `self.instance_ids` - I actually considered it - this depends on the lifecycle of the TaskInstance object, call order of the methods. So the list of instances to clean up should be cleared when XCom has communicated the result. I named the variable very purpose specific.
Would you think it would be necessary to delete the `_on_kill_instance_ids`, e.g. as part of the `post_execute` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] EC2 CreateInstance: terminate instances in on_kill [airflow]
Posted by "AchimGaedkeLynker (via GitHub)" <gi...@apache.org>.
AchimGaedkeLynker commented on code in PR #36828:
URL: https://github.com/apache/airflow/pull/36828#discussion_r1456505130
##########
airflow/providers/amazon/aws/operators/ec2.py:
##########
@@ -183,22 +183,36 @@ def execute(self, context: Context):
MaxCount=self.max_count,
**self.config,
)["Instances"]
- instance_ids = []
- for instance in instances:
- instance_ids.append(instance["InstanceId"])
- self.log.info("Created EC2 instance %s", instance["InstanceId"])
+
+ instance_ids = self._on_kill_instance_ids = [instance["InstanceId"] for instance in instances]
Review Comment:
This type of clean up in case of an error is very similar to the purpose of [context/with statements in python](https://realpython.com/python-with-statement/#the-with-statement-approach)
`on_kill` is called outside the normal execution order as part of the signal handling for `SIGTERM` - hence the extra variable only for that purpose.
Yes, one could call it `self.instance_ids` and I considered it - this depends on the lifecycle of the TaskInstance object, call order of the methods. So the list of instances to clean up should be cleared when XCom has communicated the result. I named the variable very purpose specific.
Would you think it would be necessary to delete the `_on_kill_instance_ids`, e.g. as part of the `post_execute` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org