You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "arjundevarajan (via GitHub)" <gi...@apache.org> on 2023/06/16 19:08:30 UTC
[GitHub] [superset] arjundevarajan opened a new pull request, #24432: feat: use Scarf Gateway for Superset helm charts/Docker compose downloads
arjundevarajan opened a new pull request, #24432:
URL: https://github.com/apache/superset/pull/24432
### SUMMARY
This PR updates the Superset configuration for helm charts and Docker compose to fetch Superset containers via a Scarf endpoint, so that Superset maintainers can collect basic de-identified download and adoption metrics. It does not affect where the containers are being hosted, as Scarf is only redirecting traffic back to Docker Hub.
This change was suggested by Superset maintainers in direct discussions.
### TESTING INSTRUCTIONS
To test this, download Apache Superset using the new endpoint (e.g. docker pull apachesuperset.docker.scarf.sh/apache/superset) and verify that the apache/superset container downloads without issue.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ x ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda merged pull request #24432: feat: use Scarf Gateway for Superset helm charts/Docker compose downloads
Posted by "craig-rueda (via GitHub)" <gi...@apache.org>.
craig-rueda merged PR #24432:
URL: https://github.com/apache/superset/pull/24432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on a diff in pull request #24432: feat: use Scarf Gateway for Superset helm charts/Docker compose downloads
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #24432:
URL: https://github.com/apache/superset/pull/24432#discussion_r1267139553
##########
docs/docs/installation/installing-superset-using-docker-compose.mdx:
##########
@@ -96,7 +96,14 @@ You can configure the Docker Compose environment varirables for dev and non-dev
One important variable is `SUPERSET_LOAD_EXAMPLES` which determines whether the `superset_init` container will load example data and visualizations into the database and Superset. These examples are quite helpful for most people, but probably unnecessary for experienced users. The loading process can sometimes take a few minutes and a good amount of CPU, so you may want to disable it on a resource-constrained device.
-**Note:** Users often want to connect to other databases from Superset. Currently, the easiest way to do this is to modify the `docker-compose-non-dev.yml` file and add your database as a service that the other services depend on (via `x-superset-depends-on`). Others have attempted to set `network_mode: host` on the Superset services, but these generally break the installation, because the configuration requires use of the Docker Compose DNS resolver for the service names. If you have a good solution for this, let us know!
+
+:::note
+Users often want to connect to other databases from Superset. Currently, the easiest way to do this is to modify the `docker-compose-non-dev.yml` file and add your database as a service that the other services depend on (via `x-superset-depends-on`). Others have attempted to set `network_mode: host` on the Superset services, but these generally break the installation, because the configuration requires use of the Docker Compose DNS resolver for the service names. If you have a good solution for this, let us know!
+:::
+
+:::note
+Superset uses [Scarf Gateway](https://about.scarf.sh/scarf-gateway) to collect telmetry data to better understand and support the need for patch versions of Sueprset. Scarf purges PII and provides aggregated statistics. Superset users can easily opt out of analytics in various ways documented [here](https://docs.scarf.sh/gateway/#do-not-track). However, if you wish to opt-out of this in your Docker-based installation, you can simply edit your `docker-compose.yml` or `docker-compose-non-dev.yml` file and remove `apachesuperset.docker.scarf.sh/` from the `x-superset-image` setting, so that it's simply pulling `apache/superset:${TAG:-latest-dev}`
Review Comment:
Thanks @arjundevarajan for the context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] rusackas commented on pull request #24432: feat: use Scarf Gateway for Superset helm charts/Docker compose downloads
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #24432:
URL: https://github.com/apache/superset/pull/24432#issuecomment-1636616836
@john-bodley Given that the approach had lazy consensus on the dev@ list, and the fact that there's documentation (and links to further details) in various places, I think it's safe to fo with opt-out. We can take this to the dev@ list again if it warrants further discussion, but I'm optimistic that when this makes it into a release, we'll have further changes to raise awareness about its existence and how to opt-out, on the wiki, in release notes, in the change log, etc. The main reason I'd advocate for this approach is that if it's opt-in, I suspect that we'll garner very little telemetry at all. I think this sort of telemetry is the norm in the industry at this point, and Scarf is used in other Apache projects as well. Let me know if you think this makes sense, or if this warrants widening the net on the discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] arjundevarajan commented on a diff in pull request #24432: feat: use Scarf Gateway for Superset helm charts/Docker compose downloads
Posted by "arjundevarajan (via GitHub)" <gi...@apache.org>.
arjundevarajan commented on code in PR #24432:
URL: https://github.com/apache/superset/pull/24432#discussion_r1264742404
##########
docs/docs/installation/installing-superset-using-docker-compose.mdx:
##########
@@ -96,7 +96,14 @@ You can configure the Docker Compose environment varirables for dev and non-dev
One important variable is `SUPERSET_LOAD_EXAMPLES` which determines whether the `superset_init` container will load example data and visualizations into the database and Superset. These examples are quite helpful for most people, but probably unnecessary for experienced users. The loading process can sometimes take a few minutes and a good amount of CPU, so you may want to disable it on a resource-constrained device.
-**Note:** Users often want to connect to other databases from Superset. Currently, the easiest way to do this is to modify the `docker-compose-non-dev.yml` file and add your database as a service that the other services depend on (via `x-superset-depends-on`). Others have attempted to set `network_mode: host` on the Superset services, but these generally break the installation, because the configuration requires use of the Docker Compose DNS resolver for the service names. If you have a good solution for this, let us know!
+
+:::note
+Users often want to connect to other databases from Superset. Currently, the easiest way to do this is to modify the `docker-compose-non-dev.yml` file and add your database as a service that the other services depend on (via `x-superset-depends-on`). Others have attempted to set `network_mode: host` on the Superset services, but these generally break the installation, because the configuration requires use of the Docker Compose DNS resolver for the service names. If you have a good solution for this, let us know!
+:::
+
+:::note
+Superset uses [Scarf Gateway](https://about.scarf.sh/scarf-gateway) to collect telmetry data to better understand and support the need for patch versions of Sueprset. Scarf purges PII and provides aggregated statistics. Superset users can easily opt out of analytics in various ways documented [here](https://docs.scarf.sh/gateway/#do-not-track). However, if you wish to opt-out of this in your Docker-based installation, you can simply edit your `docker-compose.yml` or `docker-compose-non-dev.yml` file and remove `apachesuperset.docker.scarf.sh/` from the `x-superset-image` setting, so that it's simply pulling `apache/superset:${TAG:-latest-dev}`
Review Comment:
Hi @john-bodley mostly chiming in to agree with @rusackas that opting for the non-default option will significantly decrease the amount of useful data that's being collected. It should be reemphasized that all of this data is de-identified no matter what, and that the ASF has approved Scarf as a verified external service provider in the past for other ASF projects (see Privacy Policy [here](https://privacy.apache.org/policies/privacy-policy-public.html#g-scarf-download-analytics)), which have deployed Scarf live to their projects for several years now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro commented on pull request #24432: feat: use Scarf Gateway for Superset helm charts/Docker compose downloads
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on PR #24432:
URL: https://github.com/apache/superset/pull/24432#issuecomment-1627087441
Hey @arjundevarajan it seems ypu need to fix lint errors, let me know if you need help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] rusackas commented on pull request #24432: feat: use Scarf Gateway for Superset helm charts/Docker compose downloads
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #24432:
URL: https://github.com/apache/superset/pull/24432#issuecomment-1627468954
OK, CI is clean. I added some documentation on this PR, so if any reviewer(s) could give that a quick 👀 it would be appreciated. We want to make sure we're transparent enough about the telemetry being added, and how to opt out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on a diff in pull request #24432: feat: use Scarf Gateway for Superset helm charts/Docker compose downloads
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #24432:
URL: https://github.com/apache/superset/pull/24432#discussion_r1260173916
##########
docs/docs/installation/installing-superset-using-docker-compose.mdx:
##########
@@ -96,7 +96,14 @@ You can configure the Docker Compose environment varirables for dev and non-dev
One important variable is `SUPERSET_LOAD_EXAMPLES` which determines whether the `superset_init` container will load example data and visualizations into the database and Superset. These examples are quite helpful for most people, but probably unnecessary for experienced users. The loading process can sometimes take a few minutes and a good amount of CPU, so you may want to disable it on a resource-constrained device.
-**Note:** Users often want to connect to other databases from Superset. Currently, the easiest way to do this is to modify the `docker-compose-non-dev.yml` file and add your database as a service that the other services depend on (via `x-superset-depends-on`). Others have attempted to set `network_mode: host` on the Superset services, but these generally break the installation, because the configuration requires use of the Docker Compose DNS resolver for the service names. If you have a good solution for this, let us know!
+
+:::note
+Users often want to connect to other databases from Superset. Currently, the easiest way to do this is to modify the `docker-compose-non-dev.yml` file and add your database as a service that the other services depend on (via `x-superset-depends-on`). Others have attempted to set `network_mode: host` on the Superset services, but these generally break the installation, because the configuration requires use of the Docker Compose DNS resolver for the service names. If you have a good solution for this, let us know!
+:::
+
+:::note
+Superset uses [Scarf Gateway](https://about.scarf.sh/scarf-gateway) to collect telmetry data to better understand and support the need for patch versions of Sueprset. Scarf purges PII and provides aggregated statistics. Superset users can easily opt out of analytics in various ways documented [here](https://docs.scarf.sh/gateway/#do-not-track). However, if you wish to opt-out of this in your Docker-based installation, you can simply edit your `docker-compose.yml` or `docker-compose-non-dev.yml` file and remove `apachesuperset.docker.scarf.sh/` from the `x-superset-image` setting, so that it's simply pulling `apache/superset:${TAG:-latest-dev}`
Review Comment:
@arjundevarajan and @rusackas would it be safer to make it opt-out by default? Granted one might not get the same scale of telemetry data, but it feels significantly less intrusive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] rusackas commented on pull request #24432: feat: use Scarf Gateway for Superset helm charts/Docker compose downloads
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #24432:
URL: https://github.com/apache/superset/pull/24432#issuecomment-1627467164
> Hey @arjundevarajan it seems you need to fix lint errors, let me know if you need help
I think it was just the helm chart version. I bumped it 🤞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org