You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/13 10:53:22 UTC
[GitHub] [airflow] tidunguyen opened a new pull request #17603: Fix MySQL database character set instruction
tidunguyen opened a new pull request #17603:
URL: https://github.com/apache/airflow/pull/17603
The currently instructed character set `utf8mb4 COLLATE utf8mb4_unicode_ci;` does not work on mysql 8.
When I do: `airflow db init` the following error occurs:
`
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1071, 'Specified key was too long; max key length is 3072 bytes')
`
Changing to this character set: `utf8 COLLATE utf8_general_ci;` solved the problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17603: Fix MySQL database character set instruction
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17603:
URL: https://github.com/apache/airflow/pull/17603#issuecomment-901122657
Do you feel you know intricacies of sqlachemy (including any migration, detection of all the cases when it is needed) etc. to do it automatcally :D ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17603: Fix MySQL database character set instruction
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17603:
URL: https://github.com/apache/airflow/pull/17603#issuecomment-898463434
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #17603: Fix MySQL database character set instruction
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #17603:
URL: https://github.com/apache/airflow/pull/17603#issuecomment-898563054
Awesome work, congrats on your first merged pull request!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #17603: Fix MySQL database character set instruction
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #17603:
URL: https://github.com/apache/airflow/pull/17603#issuecomment-901121618
Well agree this has been tactical only @uranusjr
And we actually HAVE a fix but it is purely manual configuration one and it likely could be improved in terms of "applied automatically when needed":
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#sql-engine-collation-for-ids
I think the dedicated collation just has to be "injected" into the sqlalchemy automatically when someone uses any variant of utf8mb4, It was left to discretion of the users to change it but this is a) difficult to discover b) possibly can be set automatically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #17603: Fix MySQL database character set instruction
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #17603:
URL: https://github.com/apache/airflow/pull/17603#issuecomment-901076735
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17603: Fix MySQL database character set instruction
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17603:
URL: https://github.com/apache/airflow/pull/17603#issuecomment-901925105
OK. I think I addressed it it in #17729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17603: Fix MySQL database character set instruction
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17603:
URL: https://github.com/apache/airflow/pull/17603#issuecomment-901121618
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17603: Fix MySQL database character set instruction
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17603:
URL: https://github.com/apache/airflow/pull/17603#issuecomment-901126556
Alternatively we could simply check at migrate and throw an exception "you are using utf8mb4, please set this collation_for_ids_to ...".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17603: Fix MySQL database character set instruction
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17603:
URL: https://github.com/apache/airflow/pull/17603#issuecomment-901925573
No SQLAlchemy internals needed ;).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #17603: Fix MySQL database character set instruction
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #17603:
URL: https://github.com/apache/airflow/pull/17603#issuecomment-901121618
Well agree this has been tactical only @uranusjr
And we actually HAVE a fix but it is purely manual configuration one and it likely could be improved in terms of "applied automatically when needed":
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#sql-engine-collation-for-ids
I think the dedicated collation just has to be "injected" into the sqlalchemy automatically when someone uses any variant of utf8mb4, It was left to discretion of the users to change it but this is a) difficult to discover b) possibly can be set automatically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #17603: Fix MySQL database character set instruction
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #17603:
URL: https://github.com/apache/airflow/pull/17603#issuecomment-901076735
> Changing to this character set: `utf8 COLLATE utf8_general_ci;` solved the problem
I’m very late here but is this really the correct solution? Setting a MySQL db to `utf8` is very frawned upon since this name is an alias to `utf8mb3`, which does not work with less common CJK characters and most emojis. `utf8mb4` is the much more preferred encoding. In fact, [the `utf8mb3` encoding and its `utf8` alias are actually deprecated in MySQL 8](https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-sets.html). So instead of recommending people to use `utf8mb3`, we should instead fix whatever is causing issues in the database so `utf8mb4` can be correctly applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17603: Fix MySQL database character set instruction
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17603:
URL: https://github.com/apache/airflow/pull/17603#issuecomment-901121618
Well agree this has been tactical only @uranusjr
And we actually HAVE a fix but it is purely manual configuration one and it likely could be improved in terms of "applied automatically when needed":
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#sql-engine-collation-for-ids
I think the dedicated collation just has to be "injected" into the sqlalchemy automatically when someone uses any variant of utf8, It was left to discretion of the users to change it but this is a) difficult to discover b) possibly can be set automatically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #17603: Fix MySQL database character set instruction
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #17603:
URL: https://github.com/apache/airflow/pull/17603#issuecomment-901129474
> Do you feel you know intricacies of sqlachemy
I don’t know sqlalchemy…
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #17603: Fix MySQL database character set instruction
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #17603:
URL: https://github.com/apache/airflow/pull/17603
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org