You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/30 13:43:57 UTC
[GitHub] [airflow] turbaszek opened a new pull request #9590: Improve idempotency of BigQueryInsertJobOperator
turbaszek opened a new pull request #9590:
URL: https://github.com/apache/airflow/pull/9590
Closes #8903
---
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Target Github ISSUE in description if exists
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edejong edited a comment on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
edejong edited a comment on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-666319663
Although a pretty cool solution, the difficulty I have with "reattach when the job has not finished, otherwise start a new one" is that that's quite a bit of implicit logic to follow, that may or may not be what people expect.
I would argue it's better to just keep things as predictable as possible, which means that by default it's probably best to just launch a new job. This will work fine and predictably for most use cases, and doesn't always lead to extra costs as the BigQuery cache might sometimes be used. In other words add the try number to the job id.
Then for the exceptions we can think about adding an option to the Operator to either reattach to the previous job, or cancel it. This is up to the user's specific use case scenario. So here don't use try number in the job id, then if a new job is needed make some small modification to the previous one.
That way we can start out in a predictable, simple way and more complicated use case scenarios can be added iteratively in future versions as needed. Also, because whatever happens to a previous job is explicitly set by the user it won't lead to many support questions which is what I fear otherwise. Use case scenarios vary a lot and there's really no one size fits all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek merged pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek merged pull request #9590:
URL: https://github.com/apache/airflow/pull/9590
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-669047414
> One last point, by default (force_rerun=True and reattach_states = None)if we are in scenario where there is a matching job id we throw an exception saying "consider setting force_rerun = True" This seems less than ideal.
Correct me if I'm wrong, using force_rerun will generate a unique job_id so there should be no matching job_id in Big Query?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r462182740
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1693,48 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = re.sub("\:|\-|\+", "_", context['execution_date'].isoformat())
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
Review comment:
I think we will not be able to create universal behavior that will keep idempodentency. Some users want to always execute SQL query, others want to use predetermined results, and others want to re-query only when it is backfill. All the cases sound correct and they cannot always be combined into a single operator.
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1693,48 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = re.sub("\:|\-|\+", "_", context['execution_date'].isoformat())
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
Review comment:
I think we will not be able to create universal behavior that will keep idempotency. Some users want to always execute SQL query, others want to use predetermined results, and others want to re-query only when it is backfill. All the cases sound correct and they cannot always be combined into a single operator without additional parameters.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-652306141
@albertocalderari @edejong what do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-664228615
Related #10014
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] albertocalderari commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
albertocalderari commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r448613697
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = context['execution_date'].isoformat()
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
job = hook.get_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
)
- # Get existing job and wait for it to be ready
- for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
- sleep(time_to_wait)
- job.reload()
- if job.done():
- break
+
+ if job.done() and job.error_result:
+ # The job exists and finished with an error and we are probably reruning it
+ # So we have to make a new job_id because it has to be unique
+ job_id = f"{self.job_id}_{int(time())}"
+ job = self._submit_job(hook, job_id)
+ elif not job.done():
+ # The job is still running so wait for it to be ready
+ for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
Review comment:
If you use the Google client you won't need to sleep and wait - the loop is built into the job itself
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] albertocalderari commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
albertocalderari commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r448610696
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = context['execution_date'].isoformat()
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}"
Review comment:
f"airflow_{self.dag_id}_{self.task_id}_{exec_date}" the job won't be clearable this way which might result in unexpected behavours
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r447751434
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = context['execution_date'].isoformat()
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
job = hook.get_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
)
- # Get existing job and wait for it to be ready
- for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
- sleep(time_to_wait)
- job.reload()
- if job.done():
- break
+
+ if job.done() and job.error_result:
+ # The job exists and finished with an error and we are probably reruning it
+ # So we have to make a new job_id because it has to be unique
+ job_id = f"{self.job_id}_{int(time())}"
Review comment:
```suggestion
job_id = f"{job_id}_{int(time())}"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r462256564
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1693,48 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = re.sub("\:|\-|\+", "_", context['execution_date'].isoformat())
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
Review comment:
> the query if fully rerun.
That was the original behaviour as the `job_id` was always generated by discovery API.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jaketf commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
jaketf commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r465336820
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1677,39 +1697,77 @@ def __init__(
self.project_id = project_id
self.gcp_conn_id = gcp_conn_id
self.delegate_to = delegate_to
+ self.force_rerun = force_rerun
+ self.reattach_states: Set[str] = reattach_states or set()
def prepare_template(self) -> None:
# If .json is passed then we have to read the file
if isinstance(self.configuration, str) and self.configuration.endswith('.json'):
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(
+ self,
+ hook: BigQueryHook,
+ job_id: str,
+ ) -> BigQueryJob:
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
+ @staticmethod
+ def _handle_job_error(job: BigQueryJob) -> None:
+ if job.error_result:
+ raise AirflowException(f"BigQuery job {job.job_id} failed: {job.error_result}")
+
+ def _job_id(self, context):
+ if self.force_rerun:
+ hash_base = str(uuid.uuid4())
+ else:
+ hash_base = json.dumps(self.configuration, sort_keys=True)
+
+ uniqueness_suffix = hashlib.md5(hash_base.encode()).hexdigest()
+
+ if self.job_id:
+ return f"{self.job_id}_{uniqueness_suffix}"
+
+ exec_date = re.sub(r"\:|-|\+", "_", context['execution_date'].isoformat())
+ return f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_{uniqueness_suffix}"
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ job_id = self._job_id(context)
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ job = self._submit_job(hook, job_id)
+ self._handle_job_error(job)
except Conflict:
+ # If the job already exists retrieve it
job = hook.get_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
)
- # Get existing job and wait for it to be ready
- for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
- sleep(time_to_wait)
- job.reload()
- if job.done():
- break
+ if job.state in self.reattach_states and not job.done():
+ # The job is still running so wait for it to be ready
+ job.result()
+ self._handle_job_error(job)
+ elif job.done():
+ # Same job configuration so we need force_rerun
+ raise AirflowException(
+ f"Job with id: {job_id} already exists and is in {job.state} state. If you "
+ f"want to force rerun it consider setting `force_rerun=True`."
Review comment:
```suggestion
f"want to force rerun it consider setting `force_rerun=True`."
f"Or, if you want to reattach in this scenario add {job.state} to `reattach_states`"
```
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1666,6 +1684,8 @@ def __init__(
project_id: Optional[str] = None,
location: Optional[str] = None,
job_id: Optional[str] = None,
+ force_rerun: bool = False,
Review comment:
Hmm these defaults don't seem to make sense to to me.
Why is `force_rerun = False` but `reattach_states = None` (eventually empty set) which means by default if there is an existing job id reattach candidate the user will be sad with this airflow exception telling them "should've set force_rerun to true.
We should either default force_rerun=True or have a default set of reattach states.
IMO user expectation is force_rerun=True.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] albertocalderari commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
albertocalderari commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r449065871
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = context['execution_date'].isoformat()
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
job = hook.get_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
)
- # Get existing job and wait for it to be ready
- for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
- sleep(time_to_wait)
- job.reload()
- if job.done():
- break
+
+ if job.done() and job.error_result:
+ # The job exists and finished with an error and we are probably reruning it
+ # So we have to make a new job_id because it has to be unique
+ job_id = f"{self.job_id}_{int(time())}"
+ job = self._submit_job(hook, job_id)
+ elif not job.done():
+ # The job is still running so wait for it to be ready
+ for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
Review comment:
When I hit result it polls too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edejong edited a comment on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
edejong edited a comment on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-666319663
Although a pretty cool solution, the difficulty I have with "reattach when the job has not finished, otherwise start a new one" is that that's quite a bit of implicit logic to follow, that may or may not be what people expect.
I would argue it's better to just keep things as predictable as possible, which means that by default it's probably best to just launch a new job. This will work fine and predictably for most use cases, and doesn't always lead to extra costs as the BigQuery cache might sometimes be used. In other words add the try number to the job id.
Then for the exceptions we can think about adding an option to the Operator to either reattach to the previous job, or cancel it if it's still running. This is up to the user's specific use case scenario. So here don't use try number in the job id, then if a new job is needed make some small modification to the previous one.
That way we can start out in a predictable, simple way and more complicated use case scenarios can be added iteratively in future versions as needed. Also, because whatever happens to a previous job is explicitly set by the user it won't lead to many support questions which is what I fear otherwise. Use case scenarios vary a lot and there's really no one size fits all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edejong commented on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
edejong commented on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-666319663
The difficulty I have with "reattach when the job has not finished, otherwise start a new one" is that that's quite a bit of logic to follow, that may or may not be what people expect.
I would argue it's better to just keep things as predictable as possible, which means that by default it's probably best to just launch a new job. This will work fine and predictably for most use cases, and doesn't always lead to extra costs as the BigQuery cache might sometimes be used. In other words add the try number to the job id.
Then for the exceptions we can think about adding an option to the Operator to either reattach to the previous job, or cancel it. This is up to the user's specific use case scenario. So here don't use try number in the job id, then if a new job is needed make some small modification to the previous one.
That way we can start out in a predictable, simple way and more complicated use case scenarios can be added iteratively in future versions as needed. Also, because whatever happens to a previous job is explicitly set by the user it won't lead to many support questions which is what I fear otherwise. Use case scenarios vary a lot and there's really no one size fits all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r448847105
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = context['execution_date'].isoformat()
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
job = hook.get_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
)
- # Get existing job and wait for it to be ready
- for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
- sleep(time_to_wait)
- job.reload()
- if job.done():
- break
+
+ if job.done() and job.error_result:
+ # The job exists and finished with an error and we are probably reruning it
+ # So we have to make a new job_id because it has to be unique
+ job_id = f"{self.job_id}_{int(time())}"
+ job = self._submit_job(hook, job_id)
+ elif not job.done():
+ # The job is still running so wait for it to be ready
+ for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
Review comment:
Can you provide the code? I tried to use the `job.result()` but it doesn't work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r448848371
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = context['execution_date'].isoformat()
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}"
Review comment:
This works in case we want to reattach to running job or job that succeded. The case of rerunning failed task is handled in L1727
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] albertocalderari commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
albertocalderari commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r449065662
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = context['execution_date'].isoformat()
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
job = hook.get_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
)
- # Get existing job and wait for it to be ready
- for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
- sleep(time_to_wait)
- job.reload()
- if job.done():
- break
+
+ if job.done() and job.error_result:
+ # The job exists and finished with an error and we are probably reruning it
+ # So we have to make a new job_id because it has to be unique
+ job_id = f"{self.job_id}_{int(time())}"
+ job = self._submit_job(hook, job_id)
+ elif not job.done():
+ # The job is still running so wait for it to be ready
+ for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
Review comment:
```def bq_load(config: Config, client: bigquery.Client, event: Event, source_uri: str) -> JobResultEvent:
logger.info("Loading GSheet to BQ")
job_config = LoadJobConfig(
autodetect=True,
create_disposition="CREATE_IF_NEEDED",
write_disposition="WRITE_TRUNCATE",
source_format="NEWLINE_DELIMITED_JSON"
)
logger.info("loading_table")
table_name = f"{config.project_id}.{config.dataset_name}.{event.destination_table}"
job: LoadJob = client.load_table_from_uri("gs://" + source_uri, table_name, job_config=job_config)
try:
job.result()
result_event = JobResultEvent.from_job_result_and_event(job, event)
except (GoogleCloudError, TimeoutError, ClientError) as _:
result_event = JobResultEvent.from_job_result_and_event(job, event)
logger.error("BQ Job Failed")
return result_event
``` That's one example of how I do it in an app
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jaketf edited a comment on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
jaketf edited a comment on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-666470087
Good point on predictability.
Especially for those migrating from the old BQ Operator contract (of always submit new job) it will be most predictable to have the same contract by default.
I think parameterizing a `reattach_states: Set[str]` which defaults to empty set feels really flexible this way user can say reattach to `{"PENDING", "RUNNING"}` or `{"PENDING","RUNNING", "DONE"}`.
Unfortunately, I think we'd need yet another parameter for `retry_on_reattach_to_error: bool` default to `True` (but won't have any affect unless `"DONE" in self.reattach_states).
For context I believe this is the enum of job states
- PENDING
- RUNNING
- DONE
https://cloud.google.com/bigquery/docs/reference/auditlogs/rest/Shared.Types/JobState
This notably does not distinguish between SUCCEEDED and ERROR but that requires the client code to checking the errorResult in jobstatus https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#JobStatus
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r465548486
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1666,6 +1684,8 @@ def __init__(
project_id: Optional[str] = None,
location: Optional[str] = None,
job_id: Optional[str] = None,
+ force_rerun: bool = False,
Review comment:
> IMO user expectation is force_rerun=True
I agree, that's also the old behavior
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-667657549
Very interesting discussion :).
I really like the proposal of @jaketf - with the explicitness of it (all these options are merely optimisations at the expense of predictability), but I wanted to modify it slightly. I believe there is no need to limit ourselves to the states defined in https://cloud.google.com/bigquery/docs/reference/auditlogs/rest/Shared.Types/JobState. With 4 states, we can easily define our own states:
- PENDING
- RUNNING
- SUCCESS
- FAILURE
This will be far less confusing than additional `retry_on_reattach_to_error` parameter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jaketf commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
jaketf commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r465338771
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1677,39 +1697,77 @@ def __init__(
self.project_id = project_id
self.gcp_conn_id = gcp_conn_id
self.delegate_to = delegate_to
+ self.force_rerun = force_rerun
+ self.reattach_states: Set[str] = reattach_states or set()
def prepare_template(self) -> None:
# If .json is passed then we have to read the file
if isinstance(self.configuration, str) and self.configuration.endswith('.json'):
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(
+ self,
+ hook: BigQueryHook,
+ job_id: str,
+ ) -> BigQueryJob:
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
+ @staticmethod
+ def _handle_job_error(job: BigQueryJob) -> None:
+ if job.error_result:
+ raise AirflowException(f"BigQuery job {job.job_id} failed: {job.error_result}")
+
+ def _job_id(self, context):
+ if self.force_rerun:
+ hash_base = str(uuid.uuid4())
+ else:
+ hash_base = json.dumps(self.configuration, sort_keys=True)
+
+ uniqueness_suffix = hashlib.md5(hash_base.encode()).hexdigest()
+
+ if self.job_id:
+ return f"{self.job_id}_{uniqueness_suffix}"
+
+ exec_date = re.sub(r"\:|-|\+", "_", context['execution_date'].isoformat())
+ return f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_{uniqueness_suffix}"
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ job_id = self._job_id(context)
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ job = self._submit_job(hook, job_id)
+ self._handle_job_error(job)
except Conflict:
+ # If the job already exists retrieve it
job = hook.get_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
)
- # Get existing job and wait for it to be ready
- for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
- sleep(time_to_wait)
- job.reload()
- if job.done():
- break
+ if job.state in self.reattach_states and not job.done():
Review comment:
Why `and not job.done()` it seems reasonable to want to reattach to a successful job.
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1677,39 +1697,77 @@ def __init__(
self.project_id = project_id
self.gcp_conn_id = gcp_conn_id
self.delegate_to = delegate_to
+ self.force_rerun = force_rerun
+ self.reattach_states: Set[str] = reattach_states or set()
def prepare_template(self) -> None:
# If .json is passed then we have to read the file
if isinstance(self.configuration, str) and self.configuration.endswith('.json'):
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(
+ self,
+ hook: BigQueryHook,
+ job_id: str,
+ ) -> BigQueryJob:
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
+ @staticmethod
+ def _handle_job_error(job: BigQueryJob) -> None:
+ if job.error_result:
+ raise AirflowException(f"BigQuery job {job.job_id} failed: {job.error_result}")
+
+ def _job_id(self, context):
+ if self.force_rerun:
+ hash_base = str(uuid.uuid4())
+ else:
+ hash_base = json.dumps(self.configuration, sort_keys=True)
+
+ uniqueness_suffix = hashlib.md5(hash_base.encode()).hexdigest()
+
+ if self.job_id:
+ return f"{self.job_id}_{uniqueness_suffix}"
+
+ exec_date = re.sub(r"\:|-|\+", "_", context['execution_date'].isoformat())
+ return f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_{uniqueness_suffix}"
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ job_id = self._job_id(context)
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ job = self._submit_job(hook, job_id)
+ self._handle_job_error(job)
except Conflict:
+ # If the job already exists retrieve it
job = hook.get_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
)
- # Get existing job and wait for it to be ready
- for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
- sleep(time_to_wait)
- job.reload()
- if job.done():
- break
+ if job.state in self.reattach_states and not job.done():
Review comment:
In the motivating issue https://github.com/apache/airflow/issues/8903 it seems the motivating use case was a pod dying and wanting to reattach to the job.
There is a corner case when
1. worker submits job
1. worker is dies / evicted / something
1. job succeeds
1. new worker claims this work
in this case you will raise exception rather than allowing user to reattach to a successfully completed job.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jaketf commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
jaketf commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r452557394
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1693,48 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = re.sub("\:|\-|\+", "_", context['execution_date'].isoformat())
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
Review comment:
what does this do to the the behavior for re-running a DAG for an execution date?
Would this not by default reattach to the originally succeeded job (rather than expected behavior or re-running the job)? Should this be configurable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] albertocalderari commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
albertocalderari commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r449173278
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = context['execution_date'].isoformat()
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
job = hook.get_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
)
- # Get existing job and wait for it to be ready
- for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
- sleep(time_to_wait)
- job.reload()
- if job.done():
- break
+
+ if job.done() and job.error_result:
+ # The job exists and finished with an error and we are probably reruning it
+ # So we have to make a new job_id because it has to be unique
+ job_id = f"{self.job_id}_{int(time())}"
+ job = self._submit_job(hook, job_id)
+ elif not job.done():
+ # The job is still running so wait for it to be ready
+ for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
Review comment:
In case the job is already running I still do ```job.result()``` and it polls, but without ding reload
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] albertocalderari commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
albertocalderari commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r448610390
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = context['execution_date'].isoformat()
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
job = hook.get_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
)
- # Get existing job and wait for it to be ready
- for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
- sleep(time_to_wait)
- job.reload()
- if job.done():
- break
+
+ if job.done() and job.error_result:
+ # The job exists and finished with an error and we are probably reruning it
+ # So we have to make a new job_id because it has to be unique
+ job_id = f"{self.job_id}_{int(time())}"
Review comment:
the job won't be clearable this way which might result in unexpected behavours
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] albertocalderari commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
albertocalderari commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r448610696
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = context['execution_date'].isoformat()
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}"
Review comment:
the job won't be clearable this way which might result in unexpected behavours
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jaketf commented on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
jaketf commented on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-666470087
Good point on predictability.
Especially for those migrating from the old BQ Operator contract (of always submit new job) it will be most predictable to have the same contract by default.
I think parameterizing a `reattach_states: Set[str]` which defaults to empty set feels really flexible this way user can say reattach to `{"PENDING", "RUNNING"}` or `{"PENDING","RUNNING", "DONE"}`.
Unfortunately, I think we'd need yet another parameter for `retry_on_reattach_to_error: bool` default to `True` (but won't have any affect unless `"DONE" in self.reattach_states).
For context I believe this is the enum of job states
- PENDING
- RUNNING
- DONE
https://cloud.google.com/bigquery/docs/reference/auditlogs/rest/Shared.Types/JobState
This notably does not distinguish between SUCCEEDED and ERROR but that requires checking the errorResult in jobstatus https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#JobStatus
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r452621108
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1693,48 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = re.sub("\:|\-|\+", "_", context['execution_date'].isoformat())
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
Review comment:
My question is: if job succeded why should we rerun it instead of using the existing result?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nathadfield commented on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
nathadfield commented on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-665098685
> How about using a hash of job config to create `job_id`? `{dag_id}{task_id}{exec_date}{hash(job_config)}`
I can think of scenario we could easily face where a task was successful but there was missing data in the source at the time the job was run. Rerunning in this context would mean that the `job_config` would not change if we are talking about a SQL query.
I'd argue that the job_id of each try should be unique so including `try_number` sounds like a good idea.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jaketf commented on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
jaketf commented on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-665938406
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] albertocalderari commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
albertocalderari commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r449076100
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = context['execution_date'].isoformat()
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
job = hook.get_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
)
- # Get existing job and wait for it to be ready
- for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
- sleep(time_to_wait)
- job.reload()
- if job.done():
- break
+
+ if job.done() and job.error_result:
+ # The job exists and finished with an error and we are probably reruning it
+ # So we have to make a new job_id because it has to be unique
+ job_id = f"{self.job_id}_{int(time())}"
Review comment:
I see - yet you won't be able to re-poll for this job since it uses a the current time, which is not reproducible on an "eventual" next run.
Though much better than what is now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jaketf commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
jaketf commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r462599520
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1693,48 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = re.sub("\:|\-|\+", "_", context['execution_date'].isoformat())
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
Review comment:
> I think we will not be able to create universal behavior that will keep idempotency. Some users want to always execute SQL query, others want to use predetermined results, and others want to re-query only when it is backfill. All the cases sound correct and they cannot always be combined into a single operator without additional parameters.
This is an interesting point. Perhaps we consider bringing back the deprecated BigQueryOperator (contract is always runs a new query) and than have this BigQueryInsertJobOperator (contract is create job if not succeeded exists).
IMHO we can have both behaviors in a single operator with a `force_rerun=True` parameter that controls how the job id is generated.
```python3
# choice of uuid
uniqueness_hash = hash(uuid.uuid4()) if self.force_rerun else hash(job_config)
job_id = f"{dag_id}{task_id}{exec_date}{uniqueness_hash}"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-665091094
How about using a hash of job config to create `job_id`? `{dag_id}{task_id}{exec_date}{hash(job_config)}`
This will not solve everything but we will be able to handle:
- creation of new job
- reattach to the job which is running (with same config)
- create new job in case of rerun / backfill with new changed config
We can also consider using `try_number` somewhere. Until this change, we were using random `job_id`, so the second point was not possible.
The uniqueness of `job_id` enforces a lot of limitation and I think user should be aware of that and I'm not sure if we will be able to handle 100% cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-668013078
@edejong @jaketf @potiuk @nathadfield I added changes the ``force_rerun``, ``reattach_states`` and job_id from configuration has. This operator works now in the following way:
- it calculates a unique hash of the job using job's configuration or uuid if force_rerun is True
- creates job_id in form of
[provide_job_id | airflow_{dag_id}_{task_id}_{exec_date}]_{uniqueness_suffix}
- submits a BigQuery job using the job_id
- if job with given id already exists then it tries to reattach to the job if its not done and its
state is in reattach_states. If the job is done the operator will raise AirflowException.
Using force_rerun will submit a new job every time without attaching to already existing ones.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] albertocalderari commented on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
albertocalderari commented on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-700632306
Wooohooooooo!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] albertocalderari commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
albertocalderari commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r448611939
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = context['execution_date'].isoformat()
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}"
Review comment:
I tried this before you LOL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r467797338
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1666,6 +1684,8 @@ def __init__(
project_id: Optional[str] = None,
location: Optional[str] = None,
job_id: Optional[str] = None,
+ force_rerun: bool = False,
Review comment:
```suggestion
force_rerun: bool = True,
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek edited a comment on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-668013078
@edejong @jaketf @potiuk @nathadfield I added changes the ``force_rerun``, ``reattach_states`` and job_id from configuration has. This operator works now in the following way:
- it calculates a unique hash of the job using job's configuration or uuid if force_rerun is True
- creates job_id in form of
`[provide_job_id | airflow_{dag_id}_{task_id}_{exec_date}]_{uniqueness_suffix}`
- submits a BigQuery job using the job_id
- if job with given id already exists then it tries to reattach to the job if its not done and its
state is in reattach_states. If the job is done the operator will raise AirflowException.
Using force_rerun will submit a new job every time without attaching to already existing ones.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edejong commented on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
edejong commented on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-667178180
I really like the idea of having a set of reattach_states, it's so explicit. I'd have to draw up some state diagrams to make sure sure this fulfils common use case scenarios to be sure but intuitively this sounds like the way to go to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nathadfield commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
nathadfield commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r462247317
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1693,48 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = re.sub("\:|\-|\+", "_", context['execution_date'].isoformat())
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
Review comment:
Then there should be additional parameters. IMHO, I don't think it is controversial to suggest that, when the status of a BQ task is cleared, the default behaviour should be that the query if fully rerun. Other cases sound like less common scenarios (in my experience at least).
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1693,48 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = re.sub("\:|\-|\+", "_", context['execution_date'].isoformat())
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
Review comment:
Indeed. I'd say that should be the default going forward too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r465549860
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1677,39 +1697,77 @@ def __init__(
self.project_id = project_id
self.gcp_conn_id = gcp_conn_id
self.delegate_to = delegate_to
+ self.force_rerun = force_rerun
+ self.reattach_states: Set[str] = reattach_states or set()
def prepare_template(self) -> None:
# If .json is passed then we have to read the file
if isinstance(self.configuration, str) and self.configuration.endswith('.json'):
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(
+ self,
+ hook: BigQueryHook,
+ job_id: str,
+ ) -> BigQueryJob:
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
+ @staticmethod
+ def _handle_job_error(job: BigQueryJob) -> None:
+ if job.error_result:
+ raise AirflowException(f"BigQuery job {job.job_id} failed: {job.error_result}")
+
+ def _job_id(self, context):
+ if self.force_rerun:
+ hash_base = str(uuid.uuid4())
+ else:
+ hash_base = json.dumps(self.configuration, sort_keys=True)
+
+ uniqueness_suffix = hashlib.md5(hash_base.encode()).hexdigest()
+
+ if self.job_id:
+ return f"{self.job_id}_{uniqueness_suffix}"
+
+ exec_date = re.sub(r"\:|-|\+", "_", context['execution_date'].isoformat())
+ return f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_{uniqueness_suffix}"
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ job_id = self._job_id(context)
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ job = self._submit_job(hook, job_id)
+ self._handle_job_error(job)
except Conflict:
+ # If the job already exists retrieve it
job = hook.get_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
)
- # Get existing job and wait for it to be ready
- for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
- sleep(time_to_wait)
- job.reload()
- if job.done():
- break
+ if job.state in self.reattach_states and not job.done():
Review comment:
I will remove the `job.done()` as this can be easily adjusted by users using `reattach_states`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jaketf edited a comment on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
jaketf edited a comment on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-665938406
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jaketf commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
jaketf commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r452971061
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1693,48 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = re.sub("\:|\-|\+", "_", context['execution_date'].isoformat())
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
Review comment:
In short, the underlying table / partition could have changed between the original run and the new run and in many cases the user will want to ensure that the result of this job is based on a fresh run of the query.
Airflow cannot know if the underlying table(s) have changed since the original run.
To give an example use case:
Imagine a streaming job in charge of inserting records to an hourly partitioned fact table and a DAG responsible for running some hourly analytics query (joins to dimension tables / aggregation). Once the processing time watermark passes the end of the partition interval original scheduled dag run runs (BigQueryInsertJobOperator task runs a query on the data at this processing time). Then late data (event time << processing time) arrives that should still go to the old partition (because partitioning is on event time). When the streaming job detects this it could use some mechanism (e.g. the REST API or an alert to a human who manually re-runs the dag for an execution date) to tell the DAG (and this BigQueryInsertJobOperator) to re-run because the original run for this interval / partition was actually against incomplete data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#issuecomment-673506737
@potiuk @mik-laj @olchas would you mind taking a look before merge?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jaketf commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
jaketf commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r465340904
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1677,39 +1697,77 @@ def __init__(
self.project_id = project_id
self.gcp_conn_id = gcp_conn_id
self.delegate_to = delegate_to
+ self.force_rerun = force_rerun
+ self.reattach_states: Set[str] = reattach_states or set()
def prepare_template(self) -> None:
# If .json is passed then we have to read the file
if isinstance(self.configuration, str) and self.configuration.endswith('.json'):
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(
+ self,
+ hook: BigQueryHook,
+ job_id: str,
+ ) -> BigQueryJob:
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
+ @staticmethod
+ def _handle_job_error(job: BigQueryJob) -> None:
+ if job.error_result:
+ raise AirflowException(f"BigQuery job {job.job_id} failed: {job.error_result}")
+
+ def _job_id(self, context):
+ if self.force_rerun:
+ hash_base = str(uuid.uuid4())
+ else:
+ hash_base = json.dumps(self.configuration, sort_keys=True)
+
+ uniqueness_suffix = hashlib.md5(hash_base.encode()).hexdigest()
+
+ if self.job_id:
+ return f"{self.job_id}_{uniqueness_suffix}"
+
+ exec_date = re.sub(r"\:|-|\+", "_", context['execution_date'].isoformat())
+ return f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_{uniqueness_suffix}"
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ job_id = self._job_id(context)
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ job = self._submit_job(hook, job_id)
+ self._handle_job_error(job)
except Conflict:
+ # If the job already exists retrieve it
job = hook.get_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
)
- # Get existing job and wait for it to be ready
- for time_to_wait in exponential_sleep_generator(initial=10, maximum=120):
- sleep(time_to_wait)
- job.reload()
- if job.done():
- break
+ if job.state in self.reattach_states and not job.done():
Review comment:
If this is intended then `reattach_states` should be more explicitly named reattach_non_terminal_states.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nathadfield commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
nathadfield commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r457894405
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1693,48 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
+ return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
- job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+ exec_date = re.sub("\:|\-|\+", "_", context['execution_date'].isoformat())
+ job_id = self.job_id or f"airflow_{self.dag_id}_{self.task_id}_{exec_date}_"
+
try:
- job = hook.insert_job(
- configuration=self.configuration,
- project_id=self.project_id,
- location=self.location,
- job_id=job_id,
- )
- # Start the job and wait for it to complete and get the result.
- job.result()
+ # Submit a new job
+ job = self._submit_job(hook, job_id)
except Conflict:
+ # If the job already exists retrieve it
Review comment:
@turbaszek I tend to agree with @jaketf. We frequently might want to rerun a DAG for a prior, successful execution date at some point in the future that would overwrite data in a partitioned BigQuery table. Typically this might occur because of a change in the business logic.
So, in my opinion, the `job_id` needs to be more unique than just `airflow_{self.dag_id}_{self.task_id}_{exec_date}_` in order to account for this.
If I clear the status of a task in a DAG, I expect that task to be fully rerun.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] albertocalderari commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
Posted by GitBox <gi...@apache.org>.
albertocalderari commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r448613980
##########
File path: airflow/providers/google/cloud/operators/bigquery.py
##########
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+ def _submit_job(self, hook: BigQueryHook, job_id: str):
+ # Submit a new job
+ job = hook.insert_job(
+ configuration=self.configuration,
+ project_id=self.project_id,
+ location=self.location,
+ job_id=job_id,
+ )
+ # Start the job and wait for it to complete and get the result.
+ job.result()
Review comment:
the job polls here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org