You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/24 16:27:00 UTC
[GitHub] [airflow] ashb opened a new pull request #12591: Actually run against the version of the DB we select in the matrix.
ashb opened a new pull request #12591:
URL: https://github.com/apache/airflow/pull/12591
Due to a bug in Breeze initialization code, we were always running
against Postgres 9.6 and MySQL 5.7, even when the matrix selected
something else.
(We were overwriting the POSTGRES_VERSION and MYSQL_VERSION environment
variables in initialization code)
I wouldn't be surprised if we now discover that we have some bugs on MySQL 8
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733142534
It's so bad on the MYSQL side, that I even do not know how to react .....
It seems that when you use Mysql8 Client and request utf8mb4 character, the database is created with :facepalm: :facepalm: :facepalm: :facepalm: :facepalm: :facepalm: :facepalm: latin1 collation....
This is what happens in our case. When we changed client in the docker image to use mysql8 client, our 5.7 database silently started to be created as latin1 database instead of utf8mb4 and it worked fine with the increased size, because in latin1 the row size is much smaller.
Mysql 8 client's utf8mb4 is turned into utf8mb4_0900_ai_ci which is not known to mysql 5.7 and it falls back to default which is latin1 in MySQL5.7.
TRUE STORY.
https://dev.mysql.com/doc/refman/8.0/en/charset-connection.html
> Why does this occur? After all, utf8mb4 is known to the 8.0 client and the 5.7 server, so both of them recognize it. To understand this behavior, it is necessary to understand that when the client tells the server which character set it wants to use, it really tells the server the default collation for that character set. Therefore, the aforementioned behavior occurs due to a combination of factors:
> The default collation for utf8mb4 differs between MySQL 5.7 and 8.0 (utf8mb4_general_ci for 5.7, utf8mb4_0900_ai_ci for 8.0).
> When the 8.0 client requests a character set of utf8mb4, what it sends to the server is the default 8.0 utf8mb4 collation; that is, the utf8mb4_0900_ai_ci.
> utf8mb4_0900_ai_ci is implemented only as of MySQL 8.0, so the 5.7 server does not recognize it.
> Because the 5.7 server does not recognize utf8mb4_0900_ai_ci, it cannot satisfy the client character set request, and falls back to its default character set and collation (latin1 and latin1_swedish_ci).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733149539
And Who needs it anyway?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733133482
Checking why
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733119858
| Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs
I saw this before with indexes. It's likely tth e are rruning utf8mb4 charset in our tests it likely causes some columns too be too big. I wonder though why it does not fail for 5.7.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733132880
The only thing that doesn't surprise me about MySQL versions is just how surprising each version is in it's behaviour :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733138756
I KNOW. It is a total :facepalm:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733141631
Or just increase the size passed to `sa.String(length=5000)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733149370
Why don't we say that we do not support utf8mb4 at all for mysql? Just utf8mb3?
I think this way we do not have to migrate anything and I think it will work just fine.
Seriously - with this problem in MySQL client, it is simply SAFER not to run utf8mb4 AT ALL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733726748
@ashb -> you can rebase this one now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733120644
The "fix" may be to make extra a TEXT rather than varchar column, that's why Mysql seems to suggest in their docs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733140667
Well atleast we didn't find any issues with Postgres13.
For MySQL I think let's just change the column to TEXT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733701403
@potiuk Order to merge PRs is #12596, #12614 then #12615 right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733133184
I start hating it for that reason too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733188433
I have no problem with limiting the possible combinations of mysql we support, so sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733143375
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil removed a comment on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
kaxil removed a comment on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733141631
Or just increase the size passed to `sa.String(length=5000)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733687725
I did a thorough check - and I run a few tests and It is actually a bit better than that. The "fallback" behavior is still there.
But utf8mb4 should continue to work as long as we set it as default. We had it before, but it has been removed when we added MYSQL8, and I am restoring it now.
Still the clients's behavior remains very bad but at least we mitigate it in tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733687725
I did a thorough check - and I run a few tests and It is actually a bit better than that. The "fallback" behavior is still there.
Bututf8mb4 should continue to work as long as we set it as default. We had it before, but it has been removed when we added MYSQL8, and I am restoring it now.
Still the clients's behavior remains very bad but at least we mitigate it in tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733142534
It's so bad on the MYSQL side, that I even do not know how to react .....
It seems that when you use Mysql8 Client and request utf8mb4 character, the database is created with :facepalm: :facepalm: :facepalm: :facepalm: :facepalm: :facepalm: :facepalm: latin1 collation....
This is what happens in our case. When we changed client in the docker image to use mysql8 client, our 5.7 database silently started to be created as latin1 database instead of utf8mb4 and it worked fine with the increased size, because in latin1 the row size is much smaller.
Mysql 8 client's utf8mb4 is turned into utf8mb4_0900_ai_ci which is not known to mysql 5.7.
TRUE STORY.
https://dev.mysql.com/doc/refman/8.0/en/charset-connection.html
> Why does this occur? After all, utf8mb4 is known to the 8.0 client and the 5.7 server, so both of them recognize it. To understand this behavior, it is necessary to understand that when the client tells the server which character set it wants to use, it really tells the server the default collation for that character set. Therefore, the aforementioned behavior occurs due to a combination of factors:
> The default collation for utf8mb4 differs between MySQL 5.7 and 8.0 (utf8mb4_general_ci for 5.7, utf8mb4_0900_ai_ci for 8.0).
> When the 8.0 client requests a character set of utf8mb4, what it sends to the server is the default 8.0 utf8mb4 collation; that is, the utf8mb4_0900_ai_ci.
> utf8mb4_0900_ai_ci is implemented only as of MySQL 8.0, so the 5.7 server does not recognize it.
> Because the 5.7 server does not recognize utf8mb4_0900_ai_ci, it cannot satisfy the client character set request, and falls back to its default character set and collation (latin1 and latin1_swedish_ci).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733134552
This table works fin on 5.7:
```
+--------------------+---------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+---------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| conn_id | varchar(250) | NO | UNI | NULL | |
| conn_type | varchar(500) | NO | | NULL | |
| host | varchar(500) | YES | | NULL | |
| schema | varchar(500) | YES | | NULL | |
| login | varchar(500) | YES | | NULL | |
| password | varchar(5000) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| extra | varchar(5000) | YES | | NULL | |
| is_encrypted | tinyint(1) | YES | | NULL | |
| is_extra_encrypted | tinyint(1) | YES | | NULL | |
| description | varchar(5000) | YES | | NULL | |
+--------------------+---------------+------+-----+---------+----------------+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #12591:
URL: https://github.com/apache/airflow/pull/12591
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733812462
In orer to make sure that it works faster I rebased it and pushed in our repo -> https://github.com/apache/airflow/pull/12621
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733188433
I have no problem with limiting the possible combinations of mysql we support, so sure.
Slight correction: let's phrase it as some form of "PR's welcome" rather than a "we won't ever support this".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733129051
> The "fix" may be to make extra a TEXT rather than varchar column, that's why Mysql seems to suggest in their docs.
Yep. The only thing that makes it really strange that it does not fail in MySQL 5.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733926907
All tests completed - just upload coverage to run.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733116243
Who tests the tester ..... Happy to take a look and fix those tests. Good that you found it @ashb
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733714089
> @potiuk Order to merge PRs is #12596, #12614 then #12615 right?
Correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733135596
I make that ~17k of length, which if we assume Mysql8 now uses 4 bytes per char pushses itjust over to 68k, where as at 3 bytes per it's 51k.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733094820
https://github.com/apache/airflow/issues/12588 for instance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733143305
So even our tests for MySQL 5.7 are wrong!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733120288
Could this be related:
> /usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py:593 Warning: (3778, "'utf8_general_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.")
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12591: Actually run against the version of the DB we select in the matrix.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12591:
URL: https://github.com/apache/airflow/pull/12591#issuecomment-733812462
In oder to make sure that it works faster I rebased it and pushed in our repo -> https://github.com/apache/airflow/pull/12621
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org