You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "mistercrunch (via GitHub)" <gi...@apache.org> on 2024/04/09 20:29:17 UTC

[PR] feat: disable chromium downloads by default in docker-compose [superset]

mistercrunch opened a new pull request, #27953:
URL: https://github.com/apache/superset/pull/27953

   ### SUMMARY
   
   When firing the `docker-compose up` command, which is pretty much the first step
   of most developer workflows, we kick off up 2 chromium downloads: one
   for PLAYWRIGHT and one for PUPPETEER by default. Those are used for optional features
   like alerts/reports/thumbnails/ and for CI.
   
   For convenience, I'm also introducing support for defining
   environment variables overrides for docker-compose in `docker/.env-local` while
   adding this to `.gitignore`, which developers can use without the
   fear/confusion around committing local envrionment specific settings to
   the repo.
   
   -----------------
   
   About the core feature in this PR here,
   this saves minutes upon firing `docker-compose up`, and reduce confusion
   around "what the heck is it doing!?", in a phase where we should just be
   pulling and starting docker images.
   
   Most developer workflows don't require either of those, and if it
   were the case, we should bake this into the docker image as opposed to
   installing during the bootstrap phase. In any case, devs can switch those on
   easily by tweaking the env vars specified in `docker/.env`
   
   #### PUPPETEER
   About PUPPETEER: in https://github.com/apache/superset/pull/22623, I found that it is used by Applitool in two GHAs:
   - (superset-applitool-cypress)[https://github.com/apache/superset/blob/master/.github/workflows/superset-applitool-cypress.yml]
   - (superset-applitools-storybook)[https://github.com/apache/superset/blob/master/.github/workflows/superset-applitools-storybook.yml]
   
   Seems PUPPETEER is probably never really useful locally.
   
   #### PLAYWRIGHT 
   
   About PLAYWRIGHT, this is what powers alerts and reports as well
   thumbnail-generation. This is more common, and can be flipped on
   by setting `ENABLE_PLAYWRIGHT=true` in `docker/.env-local`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat: optimize docker-compose up for faster boot time [superset]

Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #27953:
URL: https://github.com/apache/superset/pull/27953#discussion_r1561291952


##########
.github/workflows/docker.yml:
##########
@@ -21,7 +21,7 @@ jobs:
     steps:
       - id: set_matrix
         run: |
-          MATRIX_CONFIG=$(if [ "${{ github.event_name }}" == "pull_request" ]; then echo '["ci"]'; else echo '["dev", "lean", "py310", "websocket", "dockerize"]'; fi)
+          MATRIX_CONFIG=$(if [ "${{ github.event_name }}" == "pull_request" ]; then echo '["dev"]'; else echo '["dev", "lean", "py310", "websocket", "dockerize"]'; fi)

Review Comment:
   this is a bit cryptic, but the short story is we built only the `ci` layer in PRs (and a wider list on `master` push events), and I decided that building `dev` makes much more sense. I thought `ci` was all-encompassing when I first set this matrix, but turns out it's just a small layer on top of lean. Testing `dev` captures the bulk of the dockerfile.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat: disable chromium downloads by default in docker-compose [superset]

Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on PR #27953:
URL: https://github.com/apache/superset/pull/27953#issuecomment-2048579948

   @supersetbot rebase


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat: optimize docker-compose up for faster boot time [superset]

Posted by "shaoxp (via GitHub)" <gi...@apache.org>.
shaoxp commented on code in PR #27953:
URL: https://github.com/apache/superset/pull/27953#discussion_r1570141250


##########
docker-compose-image-tag.yml:
##########
@@ -33,7 +33,11 @@ services:
       - redis:/data
 
   db:
-    env_file: docker/.env
+    env_file:

Review Comment:
   these changes for env_file seems make i can not start superset following the documentation by
   `docker-compose -f docker-compose-image-tag.yml up`
   
   the error:
   ```The Compose file './docker-compose-image-tag.yml' is invalid because:
   services.db.env_file contains {"path": "docker/.env", "required": true}, which is an invalid type, it should be a string
   services.superset.env_file contains {"path": "docker/.env", "required": true}, which is an invalid type, it should be a string
   services.superset-init.env_file contains {"path": "docker/.env", "required": true}, which is an invalid type, it should be a string
   services.superset-worker.env_file contains {"path": "docker/.env", "required": true}, which is an invalid type, it should be a string
   services.superset-worker-beat.env_file contains {"path": "docker/.env", "required": true}, which is an invalid type, it should be a string```
   
   I am on Ubuntu 22.04.4 LTS 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat: disable chromium downloads by default in docker-compose [superset]

Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on PR #27953:
URL: https://github.com/apache/superset/pull/27953#issuecomment-2048582776

   @supersetbot rebase


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat: optimize docker-compose up for faster boot time [superset]

Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #27953:
URL: https://github.com/apache/superset/pull/27953#discussion_r1571005892


##########
docker-compose-image-tag.yml:
##########
@@ -33,7 +33,11 @@ services:
       - redis:/data
 
   db:
-    env_file: docker/.env
+    env_file:

Review Comment:
   hey @shaoxp , this got fixed in https://github.com/apache/superset/pull/28039 , but note that you need docker-compose 2.24+ for this config to work. I tried to lookup if there's a version of the yaml I can setup to inform users to upgrade but couldn't find anything.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat: optimize docker-compose up for faster boot time [superset]

Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #27953:
URL: https://github.com/apache/superset/pull/27953#discussion_r1561292994


##########
Dockerfile:
##########
@@ -119,9 +119,15 @@ RUN apt-get update -qq \
         libasound2 \
         libxtst6 \
         wget \
-        pkg-config \
-    # Install GeckoDriver WebDriver
-    && wget -q https://github.com/mozilla/geckodriver/releases/download/${GECKODRIVER_VERSION}/geckodriver-${GECKODRIVER_VERSION}-linux64.tar.gz -O - | tar xfz - -C /usr/local/bin \
+        git \
+        pkg-config
+
+RUN pip install playwright
+RUN playwright install-deps
+RUN playwright install chromium
+
+# Install GeckoDriver WebDriver
+RUN wget -q https://github.com/mozilla/geckodriver/releases/download/${GECKODRIVER_VERSION}/geckodriver-${GECKODRIVER_VERSION}-linux64.tar.gz -O - | tar xfz - -C /usr/local/bin \

Review Comment:
   I wonder if we could simply `playright install firefox` here instead of this line and the following. For now I decided not to touch it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat: disable chromium downloads by default in docker-compose [superset]

Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on PR #27953:
URL: https://github.com/apache/superset/pull/27953#issuecomment-2048589684

   @supersetbot rebase


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat: optimize docker-compose up for faster boot time [superset]

Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #27953:
URL: https://github.com/apache/superset/pull/27953#discussion_r1572744966


##########
docker-compose-image-tag.yml:
##########
@@ -33,7 +33,11 @@ services:
       - redis:/data
 
   db:
-    env_file: docker/.env
+    env_file:

Review Comment:
   Now I'm thinking a "check_env.py" script could be could, that script would make sure all external tools are meeting the minimum version requirements and such (npm, node, docker, docker-compose, git-lfs, ...).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat: disable chromium downloads by default in docker-compose [superset]

Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on PR #27953:
URL: https://github.com/apache/superset/pull/27953#issuecomment-2048593853

   @supersetbot rebase


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat: optimize docker-compose up for faster boot time [superset]

Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch merged PR #27953:
URL: https://github.com/apache/superset/pull/27953


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat: optimize docker-compose up for faster boot time [superset]

Posted by "shaoxp (via GitHub)" <gi...@apache.org>.
shaoxp commented on code in PR #27953:
URL: https://github.com/apache/superset/pull/27953#discussion_r1571605837


##########
docker-compose-image-tag.yml:
##########
@@ -33,7 +33,11 @@ services:
       - redis:/data
 
   db:
-    env_file: docker/.env
+    env_file:

Review Comment:
   Thanks for the info. i am able to start locally. 
   
   maybe you can update the document to include the required version. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org