You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "NeHaSinghTeqfocus (via GitHub)" <gi...@apache.org> on 2024/02/29 12:58:33 UTC
[PR] feat: Adding AWS S3 as notification method options for Alerts/Reports [superset]
NeHaSinghTeqfocus opened a new pull request, #27306:
URL: https://github.com/apache/superset/pull/27306
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
We are adding the implementation and usage of the AWS S3 upload feature in Apache Superset. This feature allows users to send reports directly to an AWS S3 bucket. There are three methods available for authentication: taking credentials from the user in the frontend, taking credentials from the superset_config.py file, or utilizing an AWS IAM role associated with the ECS task role.
<!-- ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF -->
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
#### Installation and Setup
- Ensure Apache Superset is installed and running.
- Ensure S3 bucket policy is configured for interacting with AWS S3.
- Configure Apache Superset to enable the AWS S3 upload feature.
- Follow the method-specific setup instructions outlined below.
#### Common Configuration for Enabling AWS S3 feature for all below method.
- User should update the superset_config.py (docker/pythonpath_dev/superset_config.py) file with following:
- Uncomment the following code in superset_config.py.
```
# AWS S3 reporting - uncomment to use
# FEATURE_FLAGS = {"ALERT_REPORTS": True, "ENABLE_AWS": True}
#ALERT_REPORTS_NOTIFICATION_DRY_RUN = False
# AWS Credentials
#AWS_ACCESS_KEY = '####'
#AWS_SECRET_KEY = '####'
```
#### Method 1: Frontend User Credentials
1. When accessing the feature in the frontend, users will be prompted to enter their AWS credentials.
2. Users should provide their AWS Access Key and the AWS Secret Access Key in the user interface itself.
3. Ensure proper IAM policies are applied to the user's credentials to grant necessary permissions for uploading to the specified S3 bucket.
#### Method 2: config.py Credentials
1. Update the config.py file in your Apache Superset installation directory.
2. Add the AWS credentials as configuration variables in superset_config.py.
3. Ensure proper IAM policies are applied to the user's credentials to grant necessary permissions for uploading to the specified S3 bucket.
#### Method 3: AWS IAM Role
1. Ensure that the ECS task running Apache Superset is associated with an IAM role.
2. Configure the IAM role's permissions to allow uploading to the desired S3 bucket.
3. Apache Superset will automatically use the IAM role associated with the ECS task for authentication when uploading to S3.
#### Usage
Once setup has been done, the AWS S3 can be used for sending reports/alert directly to the S3 bucket using following0 AWS S3 method options:
- AWS_S3_credentials
- AWS_S3_pyconfig
- AWS_S3_IAM
· Navigate to settings, then on Alerts & Reports.
· Create alert /report like we use to do with the Email or Slack.
· In the Notification Method select S3.
· Then select the available S3 method options: AWS_S3_credentials, AWS_S3_pyconfig, AWS_S3_IAM.
· Fill the required aws credentials in case of AWS_S3_credentials else just give the bucket name to use.
Like Email, we can also create the reports directly to the Aws S3 bucket from the dashboard and charts page. All three S3 method options would be available from the dashboard and charts page.
· Navigate to any of the dashboard or Chart.
· Create the report using Set up a S3 report and use any of the S3 method options available.
· Generated report can be seen in the Alerts & Reports page.
Note: If the User doesn’t enable the feature flag of ENABLE_AWS, but has ALERT_REPORTS set to True then Aws s3 option wouldn’t be available but he can still use the default Notification method from the Apache Superset i.e. of Email.
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ x ] Required feature flags:
- [ x ] Changes UI
- [ x ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ x ] Introduces new feature or API
- [ ] Removes existing feature or API
### Documentation
[This is an external link to Document for Frontend](https://teqfocus.sharepoint.com/:b:/s/ApacheSuperset/EcWisnChSd5Akaofb9LQebcBg2i2iC2W2nYSnYWX_wdJfg?e=tEg66m)
[This is an external link to Document for Backend](https://teqfocus.sharepoint.com/:b:/s/ApacheSuperset/EXG1Lb3ZwKlJvb0oBxoSc6MBK5SiSW9gKZ7q_4CcNvhYVw?e=zhQmP5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Adding AWS S3 as notification method options for Alerts/Reports [superset]
Posted by "deepak-kumar-007 (via GitHub)" <gi...@apache.org>.
deepak-kumar-007 commented on PR #27306:
URL: https://github.com/apache/superset/pull/27306#issuecomment-1980225391
@rusackas @giftig Thanks for your response, Can you please just give me a confirmation on, what is the current Apache superset version on master branch so that we can further give this information to our client ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Adding AWS S3 as notification method options for Alerts/Reports [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on PR #27306:
URL: https://github.com/apache/superset/pull/27306#issuecomment-1971106525
HI @NeHaSinghTeqfocus, thanks for your contribution. It looks like you're targeting the `2.1` release branch; can you retarget your PR to the latest `master` branch instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Adding AWS S3 as notification method options for Alerts/Reports [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #27306:
URL: https://github.com/apache/superset/pull/27306#issuecomment-1979883554
Unfortunately, we don't actively support 2.x releases anymore. In fact, it won't be long until 4.0 is out and 3.1 becomes the LTS support for patches (not features). We typically don't review/merge features onto older release branches, unless there's an opportunity to cherry-pick a fix from `master` onto a patch release (which would not contain features. There are no plans for a 2.2 release that _would_ include features.
My advice would be to upgrade to 3.1, and rebase your code so it can be PR'd into `master` and we can pick it up in releases from now on. If it were merged onto 2.x, it's unlikely then that it would ever be in a future release, since those are cut from `master`.
The other option is to fork Superset and build on the 2.1.x release branch freely. You're just subject to fork drift at that point though, and may miss out on various features or security fixes in future versions... trust me that you don't want to cherry-pick everything from master onto an old fork... it can get hairy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Adding AWS S3 as notification method options for Alerts/Reports [superset]
Posted by "NeHaSinghTeqfocus (via GitHub)" <gi...@apache.org>.
NeHaSinghTeqfocus closed pull request #27306: feat: Adding AWS S3 as notification method options for Alerts/Reports
URL: https://github.com/apache/superset/pull/27306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Adding AWS S3 as notification method options for Alerts/Reports [superset]
Posted by "deepak-kumar-007 (via GitHub)" <gi...@apache.org>.
deepak-kumar-007 commented on PR #27306:
URL: https://github.com/apache/superset/pull/27306#issuecomment-1978052934
Hi @giftig As per client we developed the code on top of version 2.1.0, so we need to raise the PR against the same. However you have mentioned to raise the PR on the master branch but it shows major code conflicts. Can you please suggest us to raise the PR for the version 2.1.0 only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Adding AWS S3 as notification method options for Alerts/Reports [superset]
Posted by "tjdetwill007 (via GitHub)" <gi...@apache.org>.
tjdetwill007 commented on PR #27306:
URL: https://github.com/apache/superset/pull/27306#issuecomment-1978051018
Hi @giftig As per client we developed the code on top of version 2.1.0, so we need to raise the PR against the same. However you have mentioned to raise the PR on the master branch but it shows major code conflicts. Can you please suggest us to raise the PR for the version 2.1.0 only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Adding AWS S3 as notification method options for Alerts/Reports [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on PR #27306:
URL: https://github.com/apache/superset/pull/27306#issuecomment-1980401661
@deepak-kumar-007 the master branch is the active development branch so it's ahead of all release versions.
You can see all releases here: https://github.com/apache/superset/releases
Currently the 4.0.0 release is being prepared and is in a pre-release stage. That would be the earliest release your changes could make it into.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org