You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2021/12/22 12:22:14 UTC
[GitHub] [superset] tweinreich opened a new issue #17848: Email Report not sending email
tweinreich opened a new issue #17848:
URL: https://github.com/apache/superset/issues/17848
First of all, thank you for Superset - it is an amazing app and I am thankful that it is open source.
I have tried to enable sending reports via email but I can't get Superset to send emails.
## Environment
- **OS:** Ubuntu 18.04.6 (Bionic Beaver)
- **Superset version:** 1.3.2
- **Celery version:** 4.4.7
## Logs
`uwsgi_superset.log`:
```shell
2021-12-22 11:20:00,568:DEBUG:cron_descriptor.GetText:Failed to find locale en_US
[pid: 9595|app: 0|req: 1/1] xx.x.x.x () {52 vars in 1502 bytes} [Wed Dec 22 11:20:00 2021] GET /api/v1/report/?q=(filters:!((col:type,opr:eq,value:Report)),order_column:name,order_direction:desc,page:0,page_size:25) => generated 2041 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 536 bytes (1 switches on core 0)
```
- it looks like the report is being generated
`celery-superset.log`:
```shell
[2021-12-22 10:02:58,579: DEBUG/MainProcess] | Worker: Starting Consumer
[2021-12-22 10:02:58,579: DEBUG/MainProcess] | Consumer: Starting Connection
[2021-12-22 10:02:58,587: INFO/MainProcess] Connected to sqla+sqlite:///celerydb.sqlite
[2021-12-22 10:02:58,587: DEBUG/MainProcess] ^-- substep ok
[2021-12-22 10:02:58,587: DEBUG/MainProcess] | Consumer: Starting Events
[2021-12-22 10:02:58,602: DEBUG/MainProcess] ^-- substep ok
[2021-12-22 10:02:58,602: DEBUG/MainProcess] | Consumer: Starting Tasks
[2021-12-22 10:02:58,704: DEBUG/MainProcess] ^-- substep ok
[2021-12-22 10:02:58,704: DEBUG/MainProcess] | Consumer: Starting Heart
[2021-12-22 10:02:58,704: DEBUG/MainProcess] ^-- substep ok
[2021-12-22 10:02:58,704: DEBUG/MainProcess] | Consumer: Starting event loop
[2021-12-22 10:02:58,705: INFO/MainProcess] superset@vagrant ready.
[2021-12-22 10:02:58,705: DEBUG/MainProcess] basic.qos: prefetch_count->4
```
`celerybeat.log`:
```shell
[2021-12-22 10:02:58,853: DEBUG/MainProcess] Setting default socket timeout to 30
[2021-12-22 10:02:58,853: INFO/MainProcess] beat: Starting...
[2021-12-22 10:02:58,882: DEBUG/MainProcess] Current schedule:
<ScheduleEntry: reports.scheduler reports.scheduler() <crontab: * * * * * (m/h/d/dM/MY)>
<ScheduleEntry: email_reports.schedule_hourly email_reports.schedule_hourly() <crontab: 1 * * * * (m/h/d/dM/MY)>
<ScheduleEntry: cache-warmup-hourly cache-warmup(strategy_name='top_n_dashboards', top_n=5, since='7 days ago') <crontab: 0 * * * * (m/h/d/dM/MY)>
<ScheduleEntry: reports.prune_log reports.prune_log() <crontab: 0 0 * * * (m/h/d/dM/MY)>
<ScheduleEntry: alerts.schedule_check alerts.schedule_check() <crontab: * * * * * (m/h/d/dM/MY)>
[2021-12-22 10:02:58,882: DEBUG/MainProcess] beat: Ticking with max interval->5.00 minutes
[2021-12-22 10:02:58,884: DEBUG/MainProcess] beat: Waking up in 1.11 seconds.
[2021-12-22 10:02:59,991: DEBUG/MainProcess] beat: Synchronizing schedule...
[2021-12-22 10:03:00,011: INFO/MainProcess] Scheduler: Sending due task reports.scheduler (reports.scheduler)
[2021-12-22 10:03:00,028: DEBUG/MainProcess] reports.scheduler sent. id->d7ee55b4-c685-435f-85d4-974b36472dd2
[2021-12-22 10:03:00,028: INFO/MainProcess] Scheduler: Sending due task alerts.schedule_check (alerts.schedule_check)
[2021-12-22 10:03:00,029: DEBUG/MainProcess] alerts.schedule_check sent. id->20708b0d-96d1-42b6-b107-65396581770e
[2021-12-22 10:03:00,029: DEBUG/MainProcess] beat: Waking up in 59.96 seconds.
[2021-12-22 10:04:00,046: INFO/MainProcess] Scheduler: Sending due task alerts.schedule_check (alerts.schedule_check)
[2021-12-22 10:04:00,048: DEBUG/MainProcess] alerts.schedule_check sent. id->bf48441b-c0ce-4cb9-9259-5dc20bc80743
[2021-12-22 10:04:00,048: INFO/MainProcess] Scheduler: Sending due task reports.scheduler (reports.scheduler)
[2021-12-22 10:04:00,049: DEBUG/MainProcess] reports.scheduler sent. id->233bc4be-0f5b-45a1-a0fd-3a980b17785f
[2021-12-22 10:04:00,052: DEBUG/MainProcess] beat: Waking up in 59.94 seconds.
```
- it looks like the report is being scheduled by Celery
I have successfully tried to login to the configured SMTP server and I was able to send an email with it manually.
Here is the output of `ps aux | grep celery`:
```shell
superset 8520 0.4 9.0 619632 183728 ? Sl 10:02 0:28 /home/superset/.virtualenvs/superset/bin/python -m celery worker --loglevel=DEBUG --pool=gevent -Ofair --concurrency=4 --logfile=/var/log/superset/celery-superset.log --pidfile=/run/celery-superset/superset.pid --hostname=superset@vagrant
superset 8576 0.0 8.7 588140 178496 ? Ssl 10:02 0:03 /home/superset/.virtualenvs/superset/bin/python /home/superset/.virtualenvs/superset/bin/celery beat -A superset.tasks.celery_app:app --workdir=/home/superset/app --pidfile=/var/run/superset/celerybeat.pid --logfile=/var/log/superset/celerybeat.log --loglevel=DEBUG --schedule=/var/run/superset/celerybeat-schedule
```
- it looks like the celery worker and celery beat are working
## Config
`superset_config.py` (only relevant parts from my current point of view):
```python
...
class CeleryConfig(object):
BROKER_URL = 'redis://localhost:6379/0'
CELERY_IMPORTS = (
'superset.sql_lab',
'superset.tasks',
)
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
CELERYD_LOG_LEVEL = 'DEBUG'
CELERYD_PREFETCH_MULTIPLIER = 10
CELERY_ACKS_LATE = True
CELERY_ANNOTATIONS = {
"tasks.add": {
"rate_limit": "10/s",
},
'sql_lab.get_sql_results': {
'rate_limit': '100/s',
},
'email_reports.send': {
'rate_limit': '1/s',
'time_limit': 120,
'soft_time_limit': 150,
'ignore_result': True,
},
}
CELERYBEAT_SCHEDULE = {
'cache-warmup-hourly': {
'task': 'cache-warmup',
'schedule': crontab(minute=0, hour='*'), # hourly
'kwargs': {
'strategy_name': 'top_n_dashboards',
'top_n': 5,
'since': '7 days ago',
},
},
'alerts.schedule_check': {
'task': 'alerts.schedule_check',
'schedule': crontab(minute='*', hour='*'),
},
'email_reports.schedule_hourly': {
'task': 'email_reports.schedule_hourly',
'schedule': crontab(minute=1, hour='*'),
},
'reports.scheduler': {
'task': 'reports.scheduler',
'schedule': crontab(minute='*', hour='*'),
},
'reports.prune_log': {
'task': 'reports.prune_log',
'schedule': crontab(minute=0, hour=0),
},
}
CELERY_TASK_PROTOCOL = 1
CELERY_CONFIG = CeleryConfig
FEATURE_FLAGS = {
"ALERT_REPORTS": True,
"DASHBOARD_NATIVE_FILTERS": True,
"DASHBOARD_CROSS_FILTERS": True,
"ENABLE_TEMPLATE_PROCESSING": True,
"ENABLE_SCHEDULED_EMAIL_REPORTS": True,
"EMAIL_NOTIFICATIONS": True
}
# see: https://superset.apache.org/docs/installation/configuring-superset#sip-15
SIP_15_ENABLED = True
ALERT_REPORTS_NOTIFICATION_DRY_RUN = False
WEBDRIVER_BASEURL = "http://superset:8888/"
# The base URL for the email report hyperlinks.
WEBDRIVER_BASEURL_USER_FRIENDLY = WEBDRIVER_BASEURL
...
```
- additionally, I configured the `MAIL_...` and `SMTP_...` keys according to the SMTP server I am using (as `MAIL_...` didn't seem to work, I added the `SMTP_...` keys with the same config)
## Thoughts, Observation & Expectation
- In the Alerts & reports administration panel, the column `Last run` is empty as well as the execution log
- It would be awesome if the SMTP server could be configured in the admin panel as well as being able to send a simple test mail (_I am wondering because I saw this feature has been removed since the old alerts & reports have been deprecated_ - this was already requested in #16694)
- The documentation for Alerts & reports is not sufficient to get it working (_in my case and obviously a lot of other cases as well:_ #12942, #13994, #10194, #9977, #7454, #13007, #16267, ...)
Since I don't see an execution log in the admin panel I think that probably there is an issue with my configuration somehow but I don't know because I am not seeing any errors except `cron_descriptor.GetText:Failed to find locale en_US` which is not relevant according to a comment in another issue.
Can somebody give me a hint what I could be doing wrong?
Thank you in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] chris-befoul commented on issue #17848: Email Report not sending email
Posted by GitBox <gi...@apache.org>.
chris-befoul commented on issue #17848:
URL: https://github.com/apache/superset/issues/17848#issuecomment-1014843858
@tweinreich thanks for the link, where were you able to access the celeryd log?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] tweinreich commented on issue #17848: Email Report not sending email
Posted by GitBox <gi...@apache.org>.
tweinreich commented on issue #17848:
URL: https://github.com/apache/superset/issues/17848#issuecomment-1014884804
@chris-befoul you're welcome. I accessed the log on the filesystem first (_I configured the path in the service file_). When I got it working, the log was finally visible in the GUI as well (_Alerts & Reports > Reports > Execution Log_).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] tweinreich commented on issue #17848: Email Report not sending email
Posted by GitBox <gi...@apache.org>.
tweinreich commented on issue #17848:
URL: https://github.com/apache/superset/issues/17848#issuecomment-1005102966
I finally got it. The following steps helped me to convince Superset to send emails:
## Service Configuration
- expose `SUPERSET_CONFIG_PATH` (pointing to the _file_ `superset_config.py`) in the `EnvironmentFile` for the systemd services
- set `Type=simple` and `Restart=on-failure` inside the services (I used `Type=forking` before)
## E-Mail Configuration
- I used the wrong combination of SMTP port, tls, starttls and ssl settings. The celeryd log was helpful because it contained warnings about the SSL issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] tweinreich closed issue #17848: Email Report not sending email
Posted by GitBox <gi...@apache.org>.
tweinreich closed issue #17848:
URL: https://github.com/apache/superset/issues/17848
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] chris-befoul commented on issue #17848: Email Report not sending email
Posted by GitBox <gi...@apache.org>.
chris-befoul commented on issue #17848:
URL: https://github.com/apache/superset/issues/17848#issuecomment-1014091182
@tweinreich I am working to incorporate email report sending with superset and was curious if I could see how your docker-compose.yml file is setup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] tweinreich commented on issue #17848: Email Report not sending email
Posted by GitBox <gi...@apache.org>.
tweinreich commented on issue #17848:
URL: https://github.com/apache/superset/issues/17848#issuecomment-1014290826
@chris-befoul I don't use Docker for this setup. I am using Ansible and some of my setup is based on [this role](https://github.com/onaio/ansible-superset). Please let me know if I can help you by providing more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org