You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kaxil (via GitHub)" <gi...@apache.org> on 2024/10/28 12:47:06 UTC
[I] Investigate batch processing for updating TI UUIDs to avoid long locks on large datasets [airflow]
kaxil opened a new issue, #43438:
URL: https://github.com/apache/airflow/issues/43438
This is part of the future work of https://github.com/apache/airflow/pull/43243 that migrate TI table's primary key to be a UUID7 string
Investigate batch processing for updating UUIDs to avoid long locks on large datasets. Maybe run with batches of 10000 rows??? -- TBD. We can also run some benchmarks once we do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate batch processing for updating TI UUIDs to avoid long locks on large datasets [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #43438:
URL: https://github.com/apache/airflow/issues/43438#issuecomment-2774401824
I've done some reading and I now find this doable. @kaxil , correct me if I am wrong here.
All I need to do is, consider a reasonably large TI table. Also, consider a reasonable-sized DB instance type(probably with 2vcpu and 8 GB RAM)(probably going to run DB on k8s pod and limit the resources).
I think time taken should be the only metric I should be keeping in mind. So, I'll start without batching and time it, then with batching, look for other optimizations and try to figure out the best way to reduce the time taken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate batch processing for updating TI UUIDs to avoid long locks on large datasets [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #43438:
URL: https://github.com/apache/airflow/issues/43438#issuecomment-2792477603
@rawwar What's the difference without batching and with batching?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate batch processing for updating TI UUIDs to avoid long locks on large datasets [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #43438:
URL: https://github.com/apache/airflow/issues/43438#issuecomment-2792547526
I'll update this on the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate batch processing for updating TI UUIDs to avoid long locks on large datasets [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #43438:
URL: https://github.com/apache/airflow/issues/43438#issuecomment-2774540100
@kaxil , one question. Do we really need to worry about the long locks? Given that this is a major upgrade, users will assume a maintenance window with downtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate batch processing for updating TI UUIDs to avoid long locks on large datasets [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #43438:
URL: https://github.com/apache/airflow/issues/43438#issuecomment-2774875875
Correct, query_timeout and just time to run the migration are key to measure and optimize
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate batch processing for updating TI UUIDs to avoid long locks on large datasets [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed issue #43438: Investigate batch processing for updating TI UUIDs to avoid long locks on large datasets
URL: https://github.com/apache/airflow/issues/43438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate batch processing for updating TI UUIDs to avoid long locks on large datasets [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #43438:
URL: https://github.com/apache/airflow/issues/43438#issuecomment-2792447966
I had issues with my laptop, and I completed setting up my new laptop last night.
Started doing some runs to note down the time taken to complete this migration. Used 4 CPU and 8Gb RAM:
As of now with the full update, here are the stats:
| Records | Time Taken | | | Batch Size |
|---------|-------------------------|--------|---------|------------|
| | Run-1 | Run-2 | Run-3 | |
| 750k | 58.382s | 55.702s | 55.482s| Full Update |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate batch processing for updating TI UUIDs to avoid long locks on large datasets [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #43438:
URL: https://github.com/apache/airflow/issues/43438#issuecomment-2792480409
> [@rawwar](https://github.com/rawwar) What's the difference without batching and with batching?
Running them. I'll update it as soon as I'm finished
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org