You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2021/06/14 07:08:33 UTC
[GitHub] [superset] john-bodley opened a new pull request #5449: [schema] Updating the tables schema
john-bodley opened a new pull request #5449:
URL: https://github.com/apache/superset/pull/5449
We've noticed a number of anomalies in our database caused by ill-defined forms and/or table schema definitions. This PR resolves a number of issues related to the `tables` table including:
- Modifies the uniqueness logic defined by PR https://github.com/apache/incubator-superset/pull/3978 (which was never created in a migration) to include the schema name (previously it was only based on the database/table). This is now handled by a SQLAlchemy event listener as many dialects permit multiple NULL values for columns than can contain NULL.
- Removes the pre-add for checking for uniqueness. This is no longer required as the uniqueness constraint defines the same logic. This also only worked when adding a table and thus it was possible to have duplicates names if someone edited the record.
- Adds form validation to ensure that the schema name is not part of the table name.
- Ensures the `table_name` column is non-nullable.
Note this migration will fail if the `table_name` column is NULL. One must manually fix these records as programmatically trying to remedy these invalid records is difficult as the intent is unclear and the tables may function (from a query standpoint) if SQL is provided. The following query determines which records are problematic:
```
SELECT
*
FROM
tables
WHERE
table_name IS NULL
```
Finally this migration _will_ fail if the `tables` table is corrupt in terms of the uniqueness. One **must** manually consolidate duplicate or non-valid records given there's no way of programmatically removing invalid records. The following query determines whether there are duplicates:
```
SELECT
database_id,
`schema`,
table_name,
COUNT(1)
FROM
tables
GROUP BY
1, 2, 3
HAVING
COUNT(1) > 1
ORDER BY
COUNT(1) DESC
```
Note this PR is gated by https://github.com/apache/incubator-superset/pull/5445 and https://github.com/apache/incubator-superset/pull/7084 which ensure that empty strings associated with form-data wont persist in the database and is necessary for ensuring that the relevant entries are non-NULL.
to: @fabianmenges @graceguo-supercat @michellethomas @mistercrunch @timifasubaa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-797770240
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-857139211
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-806059401
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] junlincc closed pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
junlincc closed pull request #5449:
URL: https://github.com/apache/superset/pull/5449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-884393522
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-885280175
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] junlincc closed pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
junlincc closed pull request #5449:
URL: https://github.com/apache/superset/pull/5449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] nytai commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
nytai commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-859194756
i'm pretty surprised it hasn't fallen more out of sync :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley closed pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
john-bodley closed pull request #5449:
URL: https://github.com/apache/superset/pull/5449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-810558163
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-820731675
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-881071021
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-816373614
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-818969887
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-841560252
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] srinify commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
srinify commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-770388815
@john-bodley hey John, is this PR still relevant? I was hoping to go through older PR's and issues and do some gardening. 🌻
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-820492092
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-880998217
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
john-bodley commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-887639018
Closing in favor of https://github.com/apache/superset/pull/15909.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-882958203
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-883792299
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-878023595
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-829613258
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-831230350
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
john-bodley commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-770426091
Sadly yes. I think we never resolved how to implement the ideal solution for the various Superset metadata databases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-884493309
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-882958203
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #5449: [schema] Updating the tables schema
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #5449:
URL: https://github.com/apache/superset/pull/5449#issuecomment-882958203
⚠️ @john-bodley Your base branch `master` has just also updated `superset/migrations`.
❗ **Please consider rebasing your branch to avoid db migration conflicts.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org