You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/02/13 10:28:30 UTC
[GitHub] [superset] kzosabe opened a new pull request #18694: feat(bigquery): add support for query cost estimate
kzosabe opened a new pull request #18694:
URL: https://github.com/apache/superset/pull/18694
### SUMMARY
The ability to know in advance how many bytes a query will process is important when using BigQuery.
This PR adds the ability to estimate query cost to BigQuery integration as well as other DB systems such as postgres and presto.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
before:
after:
### TESTING INSTRUCTIONS
- Enable feature flag `ESTIMATE_QUERY_COST`
- Check `ADVANCED` > `SQL Lab` > `Enable query cost estimation` in the edit dialog in the BigQuery database connection
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [x] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [x] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] kzosabe commented on a change in pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
kzosabe commented on a change in pull request #18694:
URL: https://github.com/apache/superset/pull/18694#discussion_r805756638
##########
File path: superset/db_engine_specs/bigquery.py
##########
@@ -185,6 +185,60 @@ class BigQueryEngineSpec(BaseEngineSpec):
),
}
+ @classmethod
+ def get_allow_cost_estimate(cls, extra: Dict[str, Any]) -> bool:
+ return True
+
+ @classmethod
+ def estimate_statement_cost(
+ cls, statement: str, cursor: Any, engine: Engine
+ ) -> Dict[str, Any]:
+ try:
+ # pylint: disable=import-outside-toplevel
+ from google.cloud import bigquery
+ from google.oauth2 import service_account
+ except ImportError as ex:
+ raise Exception(
+ "Could not import libraries `google.cloud` or `google.oauth2`, "
+ "which are required to be installed in your environment in order "
+ "to estimate cost"
+ ) from ex
+
+ creds = engine.dialect.credentials_info
+ credentials = service_account.Credentials.from_service_account_info(creds)
+ client = bigquery.Client(credentials=credentials)
+ dry_run_result = client.query(
+ statement, bigquery.job.QueryJobConfig(dry_run=True)
+ )
+
+ return {
+ "Total bytes processed": dry_run_result.total_bytes_processed,
+ }
+
+ @classmethod
+ def query_cost_formatter(
+ cls, raw_cost: List[Dict[str, Any]]
+ ) -> List[Dict[str, str]]:
+ def format_bytes_str(raw_bytes: int) -> str:
+ if not isinstance(raw_bytes, int):
+ return str(raw_bytes)
+ units = ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]
+ index = 0
+ bytes = float(raw_bytes)
+ while bytes >= 1024 and index < len(units) - 1:
+ bytes /= 1024
+ index += 1
+
+ return "{:.1f}".format(bytes) + f" {units[index]}"
+
+ return [
+ {
+ k: format_bytes_str(v) if k == "Total bytes processed" else str(v)
+ for k, v in row.items()
+ }
+ for row in raw_cost
+ ]
Review comment:
I think that there are several possible patterns:
a. Based on the humanize implementation, prepare methods to pass prefixes and to_next_prefixes as parameters
It allows for common implementation and same result, but is somewhat complex to implement.
b. Provide two methods, humanize_number and humanize_bytes
The behavior of the byte count display in trino and presto changes slightly.
c. Keep a separate implementation ( or share only between trino and presto )
Which do you think is the best?
For me, any of them is OK and I'll try it.
However, it might be better to work on a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro commented on a change in pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
villebro commented on a change in pull request #18694:
URL: https://github.com/apache/superset/pull/18694#discussion_r805558957
##########
File path: superset/db_engine_specs/bigquery.py
##########
@@ -185,6 +185,60 @@ class BigQueryEngineSpec(BaseEngineSpec):
),
}
+ @classmethod
+ def get_allow_cost_estimate(cls, extra: Dict[str, Any]) -> bool:
+ return True
+
+ @classmethod
+ def estimate_statement_cost(
+ cls, statement: str, cursor: Any, engine: Engine
+ ) -> Dict[str, Any]:
+ try:
+ # pylint: disable=import-outside-toplevel
+ from google.cloud import bigquery
+ from google.oauth2 import service_account
+ except ImportError as ex:
+ raise Exception(
+ "Could not import libraries `google.cloud` or `google.oauth2`, "
+ "which are required to be installed in your environment in order "
+ "to estimate cost"
+ ) from ex
+
+ creds = engine.dialect.credentials_info
+ credentials = service_account.Credentials.from_service_account_info(creds)
+ client = bigquery.Client(credentials=credentials)
+ dry_run_result = client.query(
+ statement, bigquery.job.QueryJobConfig(dry_run=True)
+ )
+
+ return {
+ "Total bytes processed": dry_run_result.total_bytes_processed,
+ }
+
+ @classmethod
+ def query_cost_formatter(
+ cls, raw_cost: List[Dict[str, Any]]
+ ) -> List[Dict[str, str]]:
+ def format_bytes_str(raw_bytes: int) -> str:
+ if not isinstance(raw_bytes, int):
+ return str(raw_bytes)
+ units = ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]
+ index = 0
+ bytes = float(raw_bytes)
+ while bytes >= 1024 and index < len(units) - 1:
+ bytes /= 1024
+ index += 1
+
+ return "{:.1f}".format(bytes) + f" {units[index]}"
+
+ return [
+ {
+ k: format_bytes_str(v) if k == "Total bytes processed" else str(v)
+ for k, v in row.items()
+ }
+ for row in raw_cost
+ ]
Review comment:
It seems this this logic overlaps with the `humanize` functions in the `query_cost_formatter` methods in `TrinoEngineSpec` and `PrestoEngineSpec` . I wonder if we should move `humanize` to BaseEngineSpec` so we could remove the duplication?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] kzosabe commented on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
kzosabe commented on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1058651441
@betodealmeida Hello, I've fixed the points you pointed out, could you please review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] edited a comment on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1038739351
# [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18694](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dba135c) into [master](https://codecov.io/gh/apache/superset/commit/225015fd5a94c1a92cc9b110e3e3db5cd532d47b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (225015f) will **increase** coverage by `0.05%`.
> The diff coverage is `85.71%`.
[](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18694 +/- ##
==========================================
+ Coverage 66.28% 66.33% +0.05%
==========================================
Files 1605 1619 +14
Lines 62863 62969 +106
Branches 6341 6341
==========================================
+ Hits 41666 41768 +102
- Misses 19545 19549 +4
Partials 1652 1652
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `52.23% <34.28%> (+0.09%)` | :arrow_up: |
| mysql | `81.38% <85.71%> (+0.05%)` | :arrow_up: |
| postgres | `81.43% <85.71%> (+0.05%)` | :arrow_up: |
| presto | `52.08% <34.28%> (+0.09%)` | :arrow_up: |
| python | `81.86% <85.71%> (+0.04%)` | :arrow_up: |
| sqlite | `81.11% <85.71%> (+0.05%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/base.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2Jhc2UucHk=) | `88.74% <50.00%> (ø)` | |
| [superset/db\_engine\_specs/bigquery.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2JpZ3F1ZXJ5LnB5) | `84.97% <85.71%> (+0.12%)` | :arrow_up: |
| [superset/db\_engine\_specs/postgres.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bvc3RncmVzLnB5) | `97.29% <100.00%> (+0.02%)` | :arrow_up: |
| [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `89.12% <100.00%> (ø)` | |
| [superset/db\_engine\_specs/trino.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | `71.05% <100.00%> (+0.25%)` | :arrow_up: |
| [superset/utils/pandas\_postprocessing.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nLnB5) | | |
| [superset/utils/pandas\_postprocessing/boxplot.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nL2JveHBsb3QucHk=) | `100.00% <0.00%> (ø)` | |
| [superset/utils/pandas\_postprocessing/aggregate.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nL2FnZ3JlZ2F0ZS5weQ==) | `90.90% <0.00%> (ø)` | |
| [superset/utils/pandas\_postprocessing/diff.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nL2RpZmYucHk=) | `100.00% <0.00%> (ø)` | |
| [...perset/utils/pandas\_postprocessing/contribution.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nL2NvbnRyaWJ1dGlvbi5weQ==) | `100.00% <0.00%> (ø)` | |
| ... and [11 more](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [225015f...dba135c](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] kzosabe commented on a change in pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
kzosabe commented on a change in pull request #18694:
URL: https://github.com/apache/superset/pull/18694#discussion_r805313408
##########
File path: superset/db_engine_specs/bigquery.py
##########
@@ -185,6 +187,49 @@ class BigQueryEngineSpec(BaseEngineSpec):
),
}
+ @classmethod
+ def get_allow_cost_estimate(cls, extra: Dict[str, Any]) -> bool:
+ return True
+
+ @classmethod
+ def estimate_statement_cost(
+ cls, statement: str, cursor: Any, engine: Engine
Review comment:
The only way to estimate the cost in advance in BigQuery is to run the query with dry_run, and since this is not possible with only cursor, I add engine as an argument.
Another way to handle bigquery.Client directly is to configure sqlalchemy to pass the dryrun parameter when creating the connection, but this seems to be more complicated...
https://github.com/googleapis/python-bigquery-sqlalchemy#connection-string-parameters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] edited a comment on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1038739351
# [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18694](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dba135c) into [master](https://codecov.io/gh/apache/superset/commit/225015fd5a94c1a92cc9b110e3e3db5cd532d47b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (225015f) will **increase** coverage by `0.03%`.
> The diff coverage is `85.71%`.
[](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18694 +/- ##
==========================================
+ Coverage 66.28% 66.31% +0.03%
==========================================
Files 1605 1619 +14
Lines 62863 62969 +106
Branches 6341 6341
==========================================
+ Hits 41666 41756 +90
- Misses 19545 19561 +16
Partials 1652 1652
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `52.23% <34.28%> (+0.09%)` | :arrow_up: |
| mysql | `81.38% <85.71%> (+0.05%)` | :arrow_up: |
| postgres | `81.43% <85.71%> (+0.05%)` | :arrow_up: |
| presto | `52.08% <34.28%> (+0.09%)` | :arrow_up: |
| python | `81.82% <85.71%> (+0.01%)` | :arrow_up: |
| sqlite | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/base.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2Jhc2UucHk=) | `88.38% <50.00%> (-0.37%)` | :arrow_down: |
| [superset/db\_engine\_specs/bigquery.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2JpZ3F1ZXJ5LnB5) | `84.97% <85.71%> (+0.12%)` | :arrow_up: |
| [superset/db\_engine\_specs/postgres.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bvc3RncmVzLnB5) | `97.29% <100.00%> (+0.02%)` | :arrow_up: |
| [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `89.12% <100.00%> (ø)` | |
| [superset/db\_engine\_specs/trino.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | `71.05% <100.00%> (+0.25%)` | :arrow_up: |
| [superset/db\_engine\_specs/sqlite.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3NxbGl0ZS5weQ==) | `91.89% <0.00%> (-5.41%)` | :arrow_down: |
| [superset/connectors/sqla/utils.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29ubmVjdG9ycy9zcWxhL3V0aWxzLnB5) | `88.23% <0.00%> (-3.93%)` | :arrow_down: |
| [superset/utils/celery.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvY2VsZXJ5LnB5) | `86.20% <0.00%> (-3.45%)` | :arrow_down: |
| [superset/result\_set.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvcmVzdWx0X3NldC5weQ==) | `96.77% <0.00%> (-1.62%)` | :arrow_down: |
| [superset/databases/commands/test\_connection.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YWJhc2VzL2NvbW1hbmRzL3Rlc3RfY29ubmVjdGlvbi5weQ==) | `98.57% <0.00%> (-1.43%)` | :arrow_down: |
| ... and [19 more](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [225015f...dba135c](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] kzosabe commented on a change in pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
kzosabe commented on a change in pull request #18694:
URL: https://github.com/apache/superset/pull/18694#discussion_r805754941
##########
File path: superset/db_engine_specs/bigquery.py
##########
@@ -185,6 +185,60 @@ class BigQueryEngineSpec(BaseEngineSpec):
),
}
+ @classmethod
+ def get_allow_cost_estimate(cls, extra: Dict[str, Any]) -> bool:
+ return True
+
+ @classmethod
+ def estimate_statement_cost(
+ cls, statement: str, cursor: Any, engine: Engine
+ ) -> Dict[str, Any]:
+ try:
+ # pylint: disable=import-outside-toplevel
+ from google.cloud import bigquery
+ from google.oauth2 import service_account
+ except ImportError as ex:
+ raise Exception(
+ "Could not import libraries `google.cloud` or `google.oauth2`, "
+ "which are required to be installed in your environment in order "
+ "to estimate cost"
+ ) from ex
+
+ creds = engine.dialect.credentials_info
+ credentials = service_account.Credentials.from_service_account_info(creds)
+ client = bigquery.Client(credentials=credentials)
+ dry_run_result = client.query(
+ statement, bigquery.job.QueryJobConfig(dry_run=True)
+ )
+
+ return {
+ "Total bytes processed": dry_run_result.total_bytes_processed,
+ }
+
+ @classmethod
+ def query_cost_formatter(
+ cls, raw_cost: List[Dict[str, Any]]
+ ) -> List[Dict[str, str]]:
+ def format_bytes_str(raw_bytes: int) -> str:
+ if not isinstance(raw_bytes, int):
+ return str(raw_bytes)
+ units = ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]
+ index = 0
+ bytes = float(raw_bytes)
+ while bytes >= 1024 and index < len(units) - 1:
+ bytes /= 1024
+ index += 1
+
+ return "{:.1f}".format(bytes) + f" {units[index]}"
+
+ return [
+ {
+ k: format_bytes_str(v) if k == "Total bytes processed" else str(v)
+ for k, v in row.items()
+ }
+ for row in raw_cost
+ ]
Review comment:
Thanks for the review!
> so we could remove the duplication?
It is possible that it is better to go DRY, but there are some things to consider somewhat.
The intent of this implementation was to be consistent with the official UI provided by BigQuery, both in KiB notation and to the first decimal place.
<img width="286" alt="スクリーンショット 2022-02-14 20 32 57" src="https://user-images.githubusercontent.com/1046476/153856821-3b8f00af-0f99-467b-afac-da850013e04f.png">
In particular, the current presto and trino implementations divide by 1000 instead of 1024, which is a problem.
There will be a small difference between the number of predicted bytes in BigQuery and the number of predicted bytes in superset.
I would like to avoid using the current humanize implementation as is, because this would cause confusion for users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] edited a comment on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1038739351
# [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18694](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9e36f0f) into [master](https://codecov.io/gh/apache/superset/commit/225015fd5a94c1a92cc9b110e3e3db5cd532d47b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (225015f) will **increase** coverage by `0.06%`.
> The diff coverage is `91.22%`.
[](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18694 +/- ##
==========================================
+ Coverage 66.28% 66.34% +0.06%
==========================================
Files 1605 1619 +14
Lines 62863 63012 +149
Branches 6341 6341
==========================================
+ Hits 41666 41805 +139
- Misses 19545 19555 +10
Partials 1652 1652
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `52.22% <24.56%> (+0.08%)` | :arrow_up: |
| mysql | `?` | |
| postgres | `81.52% <91.22%> (+0.14%)` | :arrow_up: |
| presto | `52.07% <24.56%> (+0.08%)` | :arrow_up: |
| python | `81.87% <91.22%> (+0.05%)` | :arrow_up: |
| sqlite | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/trino.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | `79.41% <50.00%> (+8.61%)` | :arrow_up: |
| [superset/db\_engine\_specs/base.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2Jhc2UucHk=) | `88.46% <91.30%> (-0.29%)` | :arrow_down: |
| [superset/db\_engine\_specs/bigquery.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2JpZ3F1ZXJ5LnB5) | `87.02% <96.00%> (+2.17%)` | :arrow_up: |
| [superset/db\_engine\_specs/postgres.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bvc3RncmVzLnB5) | `97.29% <100.00%> (+0.02%)` | :arrow_up: |
| [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `89.27% <100.00%> (+0.14%)` | :arrow_up: |
| [superset/common/utils/dataframe\_utils.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29tbW9uL3V0aWxzL2RhdGFmcmFtZV91dGlscy5weQ==) | `85.71% <0.00%> (-7.15%)` | :arrow_down: |
| [superset/db\_engine\_specs/sqlite.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3NxbGl0ZS5weQ==) | `91.89% <0.00%> (-5.41%)` | :arrow_down: |
| [superset/connectors/sqla/utils.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29ubmVjdG9ycy9zcWxhL3V0aWxzLnB5) | `88.23% <0.00%> (-3.93%)` | :arrow_down: |
| [superset/db\_engine\_specs/mysql.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL215c3FsLnB5) | `93.97% <0.00%> (-3.62%)` | :arrow_down: |
| [superset/reports/commands/log\_prune.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvcmVwb3J0cy9jb21tYW5kcy9sb2dfcHJ1bmUucHk=) | `85.71% <0.00%> (-3.58%)` | :arrow_down: |
| ... and [45 more](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [225015f...9e36f0f](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] edited a comment on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1038739351
# [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18694](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dba135c) into [master](https://codecov.io/gh/apache/superset/commit/225015fd5a94c1a92cc9b110e3e3db5cd532d47b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (225015f) will **increase** coverage by `0.05%`.
> The diff coverage is `85.71%`.
> :exclamation: Current head dba135c differs from pull request most recent head 9e36f0f. Consider uploading reports for the commit 9e36f0f to get more accurate results
[](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18694 +/- ##
==========================================
+ Coverage 66.28% 66.33% +0.05%
==========================================
Files 1605 1619 +14
Lines 62863 62969 +106
Branches 6341 6341
==========================================
+ Hits 41666 41768 +102
- Misses 19545 19549 +4
Partials 1652 1652
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `52.23% <34.28%> (+0.09%)` | :arrow_up: |
| mysql | `81.38% <85.71%> (+0.05%)` | :arrow_up: |
| postgres | `81.43% <85.71%> (+0.05%)` | :arrow_up: |
| presto | `52.08% <34.28%> (+0.09%)` | :arrow_up: |
| python | `81.86% <85.71%> (+0.04%)` | :arrow_up: |
| sqlite | `81.11% <85.71%> (+0.05%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/base.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2Jhc2UucHk=) | `88.74% <50.00%> (ø)` | |
| [superset/db\_engine\_specs/bigquery.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2JpZ3F1ZXJ5LnB5) | `84.97% <85.71%> (+0.12%)` | :arrow_up: |
| [superset/db\_engine\_specs/postgres.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bvc3RncmVzLnB5) | `97.29% <100.00%> (+0.02%)` | :arrow_up: |
| [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `89.12% <100.00%> (ø)` | |
| [superset/db\_engine\_specs/trino.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | `71.05% <100.00%> (+0.25%)` | :arrow_up: |
| [superset/utils/pandas\_postprocessing.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nLnB5) | | |
| [superset/utils/pandas\_postprocessing/\_\_init\_\_.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nL19faW5pdF9fLnB5) | `100.00% <0.00%> (ø)` | |
| [superset/utils/pandas\_postprocessing/sort.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nL3NvcnQucHk=) | `100.00% <0.00%> (ø)` | |
| [...perset/utils/pandas\_postprocessing/contribution.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nL2NvbnRyaWJ1dGlvbi5weQ==) | `100.00% <0.00%> (ø)` | |
| [superset/utils/pandas\_postprocessing/rolling.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nL3JvbGxpbmcucHk=) | `92.50% <0.00%> (ø)` | |
| ... and [11 more](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [225015f...9e36f0f](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] kzosabe commented on a change in pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
kzosabe commented on a change in pull request #18694:
URL: https://github.com/apache/superset/pull/18694#discussion_r805313408
##########
File path: superset/db_engine_specs/bigquery.py
##########
@@ -185,6 +187,49 @@ class BigQueryEngineSpec(BaseEngineSpec):
),
}
+ @classmethod
+ def get_allow_cost_estimate(cls, extra: Dict[str, Any]) -> bool:
+ return True
+
+ @classmethod
+ def estimate_statement_cost(
+ cls, statement: str, cursor: Any, engine: Engine
Review comment:
The only way to estimate the cost in advance in BigQuery is to run the query with dry_run, and since this is not possible with only cursor, I add engine as an argument.
Another way to handle bigquery.Client directly is to configure sqlalchemy to pass the dryrun parameter when creating the connection, but this seems to be more complicated...
https://github.com/googleapis/python-bigquery-sqlalchemy#connection-string-parameters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] kzosabe commented on a change in pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
kzosabe commented on a change in pull request #18694:
URL: https://github.com/apache/superset/pull/18694#discussion_r810434572
##########
File path: superset/db_engine_specs/base.py
##########
@@ -1425,6 +1430,33 @@ def cancel_query( # pylint: disable=unused-argument
def parse_sql(cls, sql: str) -> List[str]:
return [str(s).strip(" ;") for s in sqlparse.parse(sql)]
+ @classmethod
+ def _humanize(cls, value: Any, suffix: str, category: Optional[str] = None) -> str:
Review comment:
If there could be an input like (1000, "", "dollars") and an output like "$1,000", there would be more categories
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] kzosabe commented on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
kzosabe commented on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1070887497
@betodealmeida @villebro I've been waiting for a review for about 3 weeks since last update, I'd love to get some feedback on whether something is wrong with my PR or reviewers are busy.
If reviewers are busy, It's no problem and I'll wait as long as needed. This is my first OSS commit, so I would like your advice if there is anything that is not good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro commented on a change in pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
villebro commented on a change in pull request #18694:
URL: https://github.com/apache/superset/pull/18694#discussion_r805799219
##########
File path: superset/db_engine_specs/bigquery.py
##########
@@ -185,6 +185,60 @@ class BigQueryEngineSpec(BaseEngineSpec):
),
}
+ @classmethod
+ def get_allow_cost_estimate(cls, extra: Dict[str, Any]) -> bool:
+ return True
+
+ @classmethod
+ def estimate_statement_cost(
+ cls, statement: str, cursor: Any, engine: Engine
+ ) -> Dict[str, Any]:
+ try:
+ # pylint: disable=import-outside-toplevel
+ from google.cloud import bigquery
+ from google.oauth2 import service_account
+ except ImportError as ex:
+ raise Exception(
+ "Could not import libraries `google.cloud` or `google.oauth2`, "
+ "which are required to be installed in your environment in order "
+ "to estimate cost"
+ ) from ex
+
+ creds = engine.dialect.credentials_info
+ credentials = service_account.Credentials.from_service_account_info(creds)
+ client = bigquery.Client(credentials=credentials)
+ dry_run_result = client.query(
+ statement, bigquery.job.QueryJobConfig(dry_run=True)
+ )
+
+ return {
+ "Total bytes processed": dry_run_result.total_bytes_processed,
+ }
+
+ @classmethod
+ def query_cost_formatter(
+ cls, raw_cost: List[Dict[str, Any]]
+ ) -> List[Dict[str, str]]:
+ def format_bytes_str(raw_bytes: int) -> str:
+ if not isinstance(raw_bytes, int):
+ return str(raw_bytes)
+ units = ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]
+ index = 0
+ bytes = float(raw_bytes)
+ while bytes >= 1024 and index < len(units) - 1:
+ bytes /= 1024
+ index += 1
+
+ return "{:.1f}".format(bytes) + f" {units[index]}"
+
+ return [
+ {
+ k: format_bytes_str(v) if k == "Total bytes processed" else str(v)
+ for k, v in row.items()
+ }
+ for row in raw_cost
+ ]
Review comment:
Good point, being consistent with the BQ console definitely makes sense. I remember a discussion about this in the original PR where 1024 vs 1000 was debated: https://github.com/apache/superset/pull/8172#discussion_r323965087 While being consistent with the BQ console, it would feel funny to have different units in for BQ vs Presto/Trino. @betodealmeida thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] kzosabe commented on a change in pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
kzosabe commented on a change in pull request #18694:
URL: https://github.com/apache/superset/pull/18694#discussion_r810434802
##########
File path: tests/integration_tests/db_engine_specs/presto_tests.py
##########
@@ -524,7 +524,7 @@ def test_query_cost_formatter(self):
expected = [
{
"Output count": "904 M rows",
- "Output size": "354 GB",
+ "Output size": "329 GiB",
Review comment:
These two values are identical except for the units. I took care not to change any of the other outputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] edited a comment on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1038739351
# [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18694](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (35f3c6d) into [master](https://codecov.io/gh/apache/superset/commit/225015fd5a94c1a92cc9b110e3e3db5cd532d47b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (225015f) will **decrease** coverage by `0.16%`.
> The diff coverage is `85.71%`.
> :exclamation: Current head 35f3c6d differs from pull request most recent head dba135c. Consider uploading reports for the commit dba135c to get more accurate results
[](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18694 +/- ##
==========================================
- Coverage 66.28% 66.12% -0.17%
==========================================
Files 1605 1605
Lines 62863 62893 +30
Branches 6341 6341
==========================================
- Hits 41666 41585 -81
- Misses 19545 19656 +111
Partials 1652 1652
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `?` | |
| mysql | `81.33% <85.71%> (+<0.01%)` | :arrow_up: |
| postgres | `81.38% <85.71%> (+<0.01%)` | :arrow_up: |
| presto | `?` | |
| python | `81.47% <85.71%> (-0.35%)` | :arrow_down: |
| sqlite | `81.07% <85.71%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/base.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2Jhc2UucHk=) | `88.38% <50.00%> (-0.37%)` | :arrow_down: |
| [superset/db\_engine\_specs/bigquery.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2JpZ3F1ZXJ5LnB5) | `84.97% <85.71%> (+0.12%)` | :arrow_up: |
| [superset/db\_engine\_specs/postgres.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bvc3RncmVzLnB5) | `97.29% <100.00%> (+0.02%)` | :arrow_up: |
| [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `83.47% <100.00%> (-5.65%)` | :arrow_down: |
| [superset/db\_engine\_specs/trino.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | `71.05% <100.00%> (+0.25%)` | :arrow_up: |
| [superset/db\_engines/hive.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lcy9oaXZlLnB5) | `0.00% <0.00%> (-85.19%)` | :arrow_down: |
| [superset/db\_engine\_specs/hive.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2hpdmUucHk=) | `70.27% <0.00%> (-15.45%)` | :arrow_down: |
| [superset/connectors/sqla/models.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29ubmVjdG9ycy9zcWxhL21vZGVscy5weQ==) | `87.33% <0.00%> (-1.44%)` | :arrow_down: |
| ... and [2 more](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [225015f...dba135c](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] edited a comment on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1038739351
# [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18694](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dba135c) into [master](https://codecov.io/gh/apache/superset/commit/225015fd5a94c1a92cc9b110e3e3db5cd532d47b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (225015f) will **decrease** coverage by `0.01%`.
> The diff coverage is `85.71%`.
[](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18694 +/- ##
==========================================
- Coverage 66.28% 66.26% -0.02%
==========================================
Files 1605 1619 +14
Lines 62863 62969 +106
Branches 6341 6341
==========================================
+ Hits 41666 41728 +62
- Misses 19545 19589 +44
Partials 1652 1652
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `52.23% <34.28%> (+0.09%)` | :arrow_up: |
| mysql | `81.38% <85.71%> (+0.05%)` | :arrow_up: |
| postgres | `?` | |
| presto | `52.08% <34.28%> (+0.09%)` | :arrow_up: |
| python | `81.73% <85.71%> (-0.08%)` | :arrow_down: |
| sqlite | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/base.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2Jhc2UucHk=) | `88.38% <50.00%> (-0.37%)` | :arrow_down: |
| [superset/db\_engine\_specs/bigquery.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2JpZ3F1ZXJ5LnB5) | `84.97% <85.71%> (+0.12%)` | :arrow_up: |
| [superset/db\_engine\_specs/postgres.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bvc3RncmVzLnB5) | `95.49% <100.00%> (-1.78%)` | :arrow_down: |
| [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `89.12% <100.00%> (ø)` | |
| [superset/db\_engine\_specs/trino.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | `71.05% <100.00%> (+0.25%)` | :arrow_up: |
| [superset/sql\_validators/postgres.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvc3FsX3ZhbGlkYXRvcnMvcG9zdGdyZXMucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [superset/sqllab/sql\_json\_executer.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvc3FsbGFiL3NxbF9qc29uX2V4ZWN1dGVyLnB5) | `64.63% <0.00%> (-9.76%)` | :arrow_down: |
| [superset/sqllab/execution\_context\_convertor.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvc3FsbGFiL2V4ZWN1dGlvbl9jb250ZXh0X2NvbnZlcnRvci5weQ==) | `78.26% <0.00%> (-8.70%)` | :arrow_down: |
| [superset/db\_engine\_specs/sqlite.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3NxbGl0ZS5weQ==) | `91.89% <0.00%> (-5.41%)` | :arrow_down: |
| [superset/connectors/sqla/utils.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29ubmVjdG9ycy9zcWxhL3V0aWxzLnB5) | `88.23% <0.00%> (-3.93%)` | :arrow_down: |
| ... and [29 more](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [225015f...dba135c](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on a change in pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on a change in pull request #18694:
URL: https://github.com/apache/superset/pull/18694#discussion_r808364795
##########
File path: superset/db_engine_specs/bigquery.py
##########
@@ -185,6 +185,60 @@ class BigQueryEngineSpec(BaseEngineSpec):
),
}
+ @classmethod
+ def get_allow_cost_estimate(cls, extra: Dict[str, Any]) -> bool:
+ return True
+
+ @classmethod
+ def estimate_statement_cost(
+ cls, statement: str, cursor: Any, engine: Engine
+ ) -> Dict[str, Any]:
+ try:
+ # pylint: disable=import-outside-toplevel
+ from google.cloud import bigquery
+ from google.oauth2 import service_account
+ except ImportError as ex:
+ raise Exception(
+ "Could not import libraries `google.cloud` or `google.oauth2`, "
+ "which are required to be installed in your environment in order "
+ "to estimate cost"
Review comment:
Curious, wouldn't these be necessarily installed if the user has a BigQuery database connected?
##########
File path: superset/db_engine_specs/bigquery.py
##########
@@ -185,6 +185,60 @@ class BigQueryEngineSpec(BaseEngineSpec):
),
}
+ @classmethod
+ def get_allow_cost_estimate(cls, extra: Dict[str, Any]) -> bool:
+ return True
+
+ @classmethod
+ def estimate_statement_cost(
+ cls, statement: str, cursor: Any, engine: Engine
+ ) -> Dict[str, Any]:
+ try:
+ # pylint: disable=import-outside-toplevel
+ from google.cloud import bigquery
+ from google.oauth2 import service_account
+ except ImportError as ex:
+ raise Exception(
+ "Could not import libraries `google.cloud` or `google.oauth2`, "
+ "which are required to be installed in your environment in order "
+ "to estimate cost"
+ ) from ex
+
+ creds = engine.dialect.credentials_info
+ credentials = service_account.Credentials.from_service_account_info(creds)
+ client = bigquery.Client(credentials=credentials)
+ dry_run_result = client.query(
+ statement, bigquery.job.QueryJobConfig(dry_run=True)
+ )
+
+ return {
+ "Total bytes processed": dry_run_result.total_bytes_processed,
+ }
+
+ @classmethod
+ def query_cost_formatter(
+ cls, raw_cost: List[Dict[str, Any]]
+ ) -> List[Dict[str, str]]:
+ def format_bytes_str(raw_bytes: int) -> str:
+ if not isinstance(raw_bytes, int):
+ return str(raw_bytes)
+ units = ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]
+ index = 0
+ bytes = float(raw_bytes)
+ while bytes >= 1024 and index < len(units) - 1:
+ bytes /= 1024
+ index += 1
+
+ return "{:.1f}".format(bytes) + f" {units[index]}"
+
+ return [
+ {
+ k: format_bytes_str(v) if k == "Total bytes processed" else str(v)
+ for k, v in row.items()
+ }
+ for row in raw_cost
+ ]
Review comment:
I think we can change the `humanize` function in Presto (it's also duplicated in Trino) to return bytes in 1024 increments. The only reason I did 1000 is because it's also applied to row count and other parameters. This way it's consistent. Ideally we'd have a single function used by Presto, Trino, BigQuery and other engine specs.
(Note that it's also possible to overwrite the formatter function using `QUERY_COST_FORMATTERS_BY_ENGINE` in the config. We used that at Lyft to show the query cost in dollars, estimated run time, and carbon footprint.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] kzosabe commented on a change in pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
kzosabe commented on a change in pull request #18694:
URL: https://github.com/apache/superset/pull/18694#discussion_r808876857
##########
File path: superset/db_engine_specs/bigquery.py
##########
@@ -185,6 +185,60 @@ class BigQueryEngineSpec(BaseEngineSpec):
),
}
+ @classmethod
+ def get_allow_cost_estimate(cls, extra: Dict[str, Any]) -> bool:
+ return True
+
+ @classmethod
+ def estimate_statement_cost(
+ cls, statement: str, cursor: Any, engine: Engine
+ ) -> Dict[str, Any]:
+ try:
+ # pylint: disable=import-outside-toplevel
+ from google.cloud import bigquery
+ from google.oauth2 import service_account
+ except ImportError as ex:
+ raise Exception(
+ "Could not import libraries `google.cloud` or `google.oauth2`, "
+ "which are required to be installed in your environment in order "
+ "to estimate cost"
Review comment:
That's right, we can simply use import here. I'll fix this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] commented on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1038739351
# [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18694](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (35f3c6d) into [master](https://codecov.io/gh/apache/superset/commit/225015fd5a94c1a92cc9b110e3e3db5cd532d47b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (225015f) will **decrease** coverage by `0.16%`.
> The diff coverage is `85.71%`.
> :exclamation: Current head 35f3c6d differs from pull request most recent head 6c06b05. Consider uploading reports for the commit 6c06b05 to get more accurate results
[](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18694 +/- ##
==========================================
- Coverage 66.28% 66.11% -0.17%
==========================================
Files 1605 1605
Lines 62863 62893 +30
Branches 6341 6341
==========================================
- Hits 41666 41581 -85
- Misses 19545 19660 +115
Partials 1652 1652
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `?` | |
| mysql | `81.33% <85.71%> (+<0.01%)` | :arrow_up: |
| postgres | `81.37% <85.71%> (-0.01%)` | :arrow_down: |
| presto | `?` | |
| python | `81.46% <85.71%> (-0.36%)` | :arrow_down: |
| sqlite | `81.07% <85.71%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/base.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2Jhc2UucHk=) | `88.38% <50.00%> (-0.37%)` | :arrow_down: |
| [superset/db\_engine\_specs/bigquery.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2JpZ3F1ZXJ5LnB5) | `84.97% <85.71%> (+0.12%)` | :arrow_up: |
| [superset/db\_engine\_specs/postgres.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bvc3RncmVzLnB5) | `97.29% <100.00%> (+0.02%)` | :arrow_up: |
| [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `83.47% <100.00%> (-5.65%)` | :arrow_down: |
| [superset/db\_engine\_specs/trino.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | `71.05% <100.00%> (+0.25%)` | :arrow_up: |
| [superset/db\_engines/hive.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lcy9oaXZlLnB5) | `0.00% <0.00%> (-85.19%)` | :arrow_down: |
| [superset/db\_engine\_specs/hive.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2hpdmUucHk=) | `70.27% <0.00%> (-15.45%)` | :arrow_down: |
| [superset/reports/commands/log\_prune.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvcmVwb3J0cy9jb21tYW5kcy9sb2dfcHJ1bmUucHk=) | `85.71% <0.00%> (-3.58%)` | :arrow_down: |
| [superset/commands/importers/v1/utils.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29tbWFuZHMvaW1wb3J0ZXJzL3YxL3V0aWxzLnB5) | `89.13% <0.00%> (-2.18%)` | :arrow_down: |
| ... and [6 more](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [225015f...6c06b05](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] edited a comment on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1038739351
# [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18694](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dba135c) into [master](https://codecov.io/gh/apache/superset/commit/225015fd5a94c1a92cc9b110e3e3db5cd532d47b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (225015f) will **increase** coverage by `0.05%`.
> The diff coverage is `85.71%`.
[](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18694 +/- ##
==========================================
+ Coverage 66.28% 66.33% +0.05%
==========================================
Files 1605 1619 +14
Lines 62863 62969 +106
Branches 6341 6341
==========================================
+ Hits 41666 41768 +102
- Misses 19545 19549 +4
Partials 1652 1652
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `52.23% <34.28%> (+0.09%)` | :arrow_up: |
| mysql | `81.38% <85.71%> (+0.05%)` | :arrow_up: |
| postgres | `81.43% <85.71%> (+0.05%)` | :arrow_up: |
| presto | `52.08% <34.28%> (+0.09%)` | :arrow_up: |
| python | `81.86% <85.71%> (+0.04%)` | :arrow_up: |
| sqlite | `81.11% <85.71%> (+0.05%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/base.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2Jhc2UucHk=) | `88.74% <50.00%> (ø)` | |
| [superset/db\_engine\_specs/bigquery.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2JpZ3F1ZXJ5LnB5) | `84.97% <85.71%> (+0.12%)` | :arrow_up: |
| [superset/db\_engine\_specs/postgres.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bvc3RncmVzLnB5) | `97.29% <100.00%> (+0.02%)` | :arrow_up: |
| [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `89.12% <100.00%> (ø)` | |
| [superset/db\_engine\_specs/trino.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | `71.05% <100.00%> (+0.25%)` | :arrow_up: |
| [superset/utils/pandas\_postprocessing.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nLnB5) | | |
| [superset/utils/pandas\_postprocessing/\_\_init\_\_.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nL19faW5pdF9fLnB5) | `100.00% <0.00%> (ø)` | |
| [superset/utils/pandas\_postprocessing/boxplot.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nL2JveHBsb3QucHk=) | `100.00% <0.00%> (ø)` | |
| [superset/utils/pandas\_postprocessing/geography.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nL2dlb2dyYXBoeS5weQ==) | `82.85% <0.00%> (ø)` | |
| [superset/utils/pandas\_postprocessing/cum.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nL2N1bS5weQ==) | `100.00% <0.00%> (ø)` | |
| ... and [11 more](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [225015f...dba135c](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] kzosabe commented on a change in pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
kzosabe commented on a change in pull request #18694:
URL: https://github.com/apache/superset/pull/18694#discussion_r807789432
##########
File path: superset/db_engine_specs/bigquery.py
##########
@@ -185,6 +185,60 @@ class BigQueryEngineSpec(BaseEngineSpec):
),
}
+ @classmethod
+ def get_allow_cost_estimate(cls, extra: Dict[str, Any]) -> bool:
+ return True
+
+ @classmethod
+ def estimate_statement_cost(
+ cls, statement: str, cursor: Any, engine: Engine
+ ) -> Dict[str, Any]:
+ try:
+ # pylint: disable=import-outside-toplevel
+ from google.cloud import bigquery
+ from google.oauth2 import service_account
+ except ImportError as ex:
+ raise Exception(
+ "Could not import libraries `google.cloud` or `google.oauth2`, "
+ "which are required to be installed in your environment in order "
+ "to estimate cost"
+ ) from ex
+
+ creds = engine.dialect.credentials_info
+ credentials = service_account.Credentials.from_service_account_info(creds)
+ client = bigquery.Client(credentials=credentials)
+ dry_run_result = client.query(
+ statement, bigquery.job.QueryJobConfig(dry_run=True)
+ )
+
+ return {
+ "Total bytes processed": dry_run_result.total_bytes_processed,
+ }
+
+ @classmethod
+ def query_cost_formatter(
+ cls, raw_cost: List[Dict[str, Any]]
+ ) -> List[Dict[str, str]]:
+ def format_bytes_str(raw_bytes: int) -> str:
+ if not isinstance(raw_bytes, int):
+ return str(raw_bytes)
+ units = ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]
+ index = 0
+ bytes = float(raw_bytes)
+ while bytes >= 1024 and index < len(units) - 1:
+ bytes /= 1024
+ index += 1
+
+ return "{:.1f}".format(bytes) + f" {units[index]}"
+
+ return [
+ {
+ k: format_bytes_str(v) if k == "Total bytes processed" else str(v)
+ for k, v in row.items()
+ }
+ for row in raw_cost
+ ]
Review comment:
Does anyone have any suggestions?
I think the ability of estimating bytes is important, and the format is relatively unimportant, so I'm not strongly concerned about how to handle it.
I think it would be better to refactor trino and presto to KiB notation, but if there are some reasons not to do so, I'll make the BigQuery implementation as KB notation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] edited a comment on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1038739351
# [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18694](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (35f3c6d) into [master](https://codecov.io/gh/apache/superset/commit/225015fd5a94c1a92cc9b110e3e3db5cd532d47b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (225015f) will **decrease** coverage by `0.16%`.
> The diff coverage is `85.71%`.
> :exclamation: Current head 35f3c6d differs from pull request most recent head 6c06b05. Consider uploading reports for the commit 6c06b05 to get more accurate results
[](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18694 +/- ##
==========================================
- Coverage 66.28% 66.12% -0.17%
==========================================
Files 1605 1605
Lines 62863 62893 +30
Branches 6341 6341
==========================================
- Hits 41666 41585 -81
- Misses 19545 19656 +111
Partials 1652 1652
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `?` | |
| mysql | `81.33% <85.71%> (+<0.01%)` | :arrow_up: |
| postgres | `81.38% <85.71%> (+<0.01%)` | :arrow_up: |
| presto | `?` | |
| python | `81.47% <85.71%> (-0.35%)` | :arrow_down: |
| sqlite | `81.07% <85.71%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/base.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2Jhc2UucHk=) | `88.38% <50.00%> (-0.37%)` | :arrow_down: |
| [superset/db\_engine\_specs/bigquery.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2JpZ3F1ZXJ5LnB5) | `84.97% <85.71%> (+0.12%)` | :arrow_up: |
| [superset/db\_engine\_specs/postgres.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bvc3RncmVzLnB5) | `97.29% <100.00%> (+0.02%)` | :arrow_up: |
| [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `83.47% <100.00%> (-5.65%)` | :arrow_down: |
| [superset/db\_engine\_specs/trino.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | `71.05% <100.00%> (+0.25%)` | :arrow_up: |
| [superset/db\_engines/hive.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lcy9oaXZlLnB5) | `0.00% <0.00%> (-85.19%)` | :arrow_down: |
| [superset/db\_engine\_specs/hive.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2hpdmUucHk=) | `70.27% <0.00%> (-15.45%)` | :arrow_down: |
| [superset/connectors/sqla/models.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29ubmVjdG9ycy9zcWxhL21vZGVscy5weQ==) | `87.33% <0.00%> (-1.44%)` | :arrow_down: |
| ... and [2 more](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [225015f...6c06b05](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] edited a comment on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1038739351
# [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18694](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9e36f0f) into [master](https://codecov.io/gh/apache/superset/commit/225015fd5a94c1a92cc9b110e3e3db5cd532d47b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (225015f) will **increase** coverage by `0.08%`.
> The diff coverage is `91.22%`.
[](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18694 +/- ##
==========================================
+ Coverage 66.28% 66.36% +0.08%
==========================================
Files 1605 1619 +14
Lines 62863 63012 +149
Branches 6341 6341
==========================================
+ Hits 41666 41818 +152
+ Misses 19545 19542 -3
Partials 1652 1652
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `52.22% <24.56%> (+0.08%)` | :arrow_up: |
| mysql | `?` | |
| postgres | `81.52% <91.22%> (+0.14%)` | :arrow_up: |
| presto | `52.07% <24.56%> (+0.08%)` | :arrow_up: |
| python | `81.91% <91.22%> (+0.09%)` | :arrow_up: |
| sqlite | `81.22% <91.22%> (+0.15%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/trino.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | `79.41% <50.00%> (+8.61%)` | :arrow_up: |
| [superset/db\_engine\_specs/base.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2Jhc2UucHk=) | `88.98% <91.30%> (+0.23%)` | :arrow_up: |
| [superset/db\_engine\_specs/bigquery.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2JpZ3F1ZXJ5LnB5) | `87.02% <96.00%> (+2.17%)` | :arrow_up: |
| [superset/db\_engine\_specs/postgres.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bvc3RncmVzLnB5) | `97.29% <100.00%> (+0.02%)` | :arrow_up: |
| [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `89.27% <100.00%> (+0.14%)` | :arrow_up: |
| [superset/common/utils/dataframe\_utils.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29tbW9uL3V0aWxzL2RhdGFmcmFtZV91dGlscy5weQ==) | `85.71% <0.00%> (-7.15%)` | :arrow_down: |
| [superset/db\_engine\_specs/mysql.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL215c3FsLnB5) | `93.97% <0.00%> (-3.62%)` | :arrow_down: |
| [superset/reports/commands/log\_prune.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvcmVwb3J0cy9jb21tYW5kcy9sb2dfcHJ1bmUucHk=) | `85.71% <0.00%> (-3.58%)` | :arrow_down: |
| [superset/key\_value/commands/create.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2NyZWF0ZS5weQ==) | `80.95% <0.00%> (-3.05%)` | :arrow_down: |
| [superset/commands/importers/v1/utils.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29tbWFuZHMvaW1wb3J0ZXJzL3YxL3V0aWxzLnB5) | `89.13% <0.00%> (-2.18%)` | :arrow_down: |
| ... and [37 more](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [225015f...9e36f0f](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] edited a comment on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1038739351
# [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18694](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9e36f0f) into [master](https://codecov.io/gh/apache/superset/commit/225015fd5a94c1a92cc9b110e3e3db5cd532d47b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (225015f) will **increase** coverage by `0.10%`.
> The diff coverage is `91.22%`.
[](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18694 +/- ##
==========================================
+ Coverage 66.28% 66.38% +0.10%
==========================================
Files 1605 1619 +14
Lines 62863 63012 +149
Branches 6341 6341
==========================================
+ Hits 41666 41831 +165
+ Misses 19545 19529 -16
Partials 1652 1652
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `52.22% <24.56%> (+0.08%)` | :arrow_up: |
| mysql | `81.48% <91.22%> (+0.15%)` | :arrow_up: |
| postgres | `81.52% <91.22%> (+0.14%)` | :arrow_up: |
| presto | `52.07% <24.56%> (+0.08%)` | :arrow_up: |
| python | `81.95% <91.22%> (+0.13%)` | :arrow_up: |
| sqlite | `81.22% <91.22%> (+0.15%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/trino.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | `79.41% <50.00%> (+8.61%)` | :arrow_up: |
| [superset/db\_engine\_specs/base.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2Jhc2UucHk=) | `88.98% <91.30%> (+0.23%)` | :arrow_up: |
| [superset/db\_engine\_specs/bigquery.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2JpZ3F1ZXJ5LnB5) | `87.02% <96.00%> (+2.17%)` | :arrow_up: |
| [superset/db\_engine\_specs/postgres.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bvc3RncmVzLnB5) | `97.29% <100.00%> (+0.02%)` | :arrow_up: |
| [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `89.27% <100.00%> (+0.14%)` | :arrow_up: |
| [superset/reports/commands/log\_prune.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvcmVwb3J0cy9jb21tYW5kcy9sb2dfcHJ1bmUucHk=) | `85.71% <0.00%> (-3.58%)` | :arrow_down: |
| [superset/key\_value/commands/create.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2NyZWF0ZS5weQ==) | `80.95% <0.00%> (-3.05%)` | :arrow_down: |
| [superset/commands/importers/v1/utils.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29tbWFuZHMvaW1wb3J0ZXJzL3YxL3V0aWxzLnB5) | `89.13% <0.00%> (-2.18%)` | :arrow_down: |
| [superset/common/query\_object.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29tbW9uL3F1ZXJ5X29iamVjdC5weQ==) | `95.13% <0.00%> (-0.50%)` | :arrow_down: |
| [superset/reports/commands/execute.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvcmVwb3J0cy9jb21tYW5kcy9leGVjdXRlLnB5) | `91.14% <0.00%> (-0.37%)` | :arrow_down: |
| ... and [33 more](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [225015f...9e36f0f](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] edited a comment on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1038739351
# [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18694](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9e36f0f) into [master](https://codecov.io/gh/apache/superset/commit/225015fd5a94c1a92cc9b110e3e3db5cd532d47b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (225015f) will **increase** coverage by `0.11%`.
> The diff coverage is `91.22%`.
[](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18694 +/- ##
==========================================
+ Coverage 66.28% 66.39% +0.11%
==========================================
Files 1605 1619 +14
Lines 62863 63012 +149
Branches 6341 6341
==========================================
+ Hits 41666 41835 +169
+ Misses 19545 19525 -20
Partials 1652 1652
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `52.22% <24.56%> (+0.08%)` | :arrow_up: |
| mysql | `81.48% <91.22%> (+0.15%)` | :arrow_up: |
| postgres | `81.53% <91.22%> (+0.15%)` | :arrow_up: |
| presto | `52.07% <24.56%> (+0.08%)` | :arrow_up: |
| python | `81.96% <91.22%> (+0.15%)` | :arrow_up: |
| sqlite | `81.22% <91.22%> (+0.15%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/trino.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | `79.41% <50.00%> (+8.61%)` | :arrow_up: |
| [superset/db\_engine\_specs/base.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2Jhc2UucHk=) | `88.98% <91.30%> (+0.23%)` | :arrow_up: |
| [superset/db\_engine\_specs/bigquery.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2JpZ3F1ZXJ5LnB5) | `87.02% <96.00%> (+2.17%)` | :arrow_up: |
| [superset/db\_engine\_specs/postgres.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bvc3RncmVzLnB5) | `97.29% <100.00%> (+0.02%)` | :arrow_up: |
| [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `89.27% <100.00%> (+0.14%)` | :arrow_up: |
| [superset/key\_value/commands/create.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2NyZWF0ZS5weQ==) | `80.95% <0.00%> (-3.05%)` | :arrow_down: |
| [superset/key\_value/utils.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL3V0aWxzLnB5) | `100.00% <0.00%> (ø)` | |
| [superset/dashboards/filter\_state/api.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGFzaGJvYXJkcy9maWx0ZXJfc3RhdGUvYXBpLnB5) | `100.00% <0.00%> (ø)` | |
| [superset/explore/form\_data/commands/parameters.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZXhwbG9yZS9mb3JtX2RhdGEvY29tbWFuZHMvcGFyYW1ldGVycy5weQ==) | `100.00% <0.00%> (ø)` | |
| [superset/utils/pandas\_postprocessing.py](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nLnB5) | | |
| ... and [30 more](https://codecov.io/gh/apache/superset/pull/18694/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [225015f...9e36f0f](https://codecov.io/gh/apache/superset/pull/18694?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] kzosabe commented on pull request #18694: feat(bigquery): add support for query cost estimate
Posted by GitBox <gi...@apache.org>.
kzosabe commented on pull request #18694:
URL: https://github.com/apache/superset/pull/18694#issuecomment-1070887497
@betodealmeida @villebro I've been waiting for a review for about 3 weeks since last update, I'd love to get some feedback on whether something is wrong with my PR or reviewers are busy.
If reviewers are busy, It's no problem and I'll wait as long as needed. This is my first OSS commit, so I would like your advice if there is anything that is not good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org