You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/03/08 05:18:00 UTC

[GitHub] [airflow] retornam opened a new pull request #7653: [AIRFLOW-6914] Add a default robots.txt

retornam opened a new pull request #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653
 
 
   If an Airflow instance is public, Google, Bing,
   DuckDuckGo and other search engines can index it.
   If authentication has not been enabled, this can
   create a serious security issue.
   
   This fix adds a robots.txt to DENY all robot indexing,
   adds the X-Robots-Tag header and also adds the
   no-index meta tag
   Signed-off-by: Raymond Etornam <re...@users.noreply.github.com>
   
   ---
   Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   <sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#discussion_r404731596
 
 

 ##########
 File path: airflow/www/webpack.config.js
 ##########
 @@ -98,6 +98,10 @@ const config = {
       {
         test: /\.(ttf|eot|svg)(\?v=[0-9]\.[0-9]\.[0-9])?$/,
         loader: 'file-loader',
+      },{
+        test: /\.txt$/,
+        include: STATIC_DIR,
+        loader: 'file-loader',
 
 Review comment:
   This isn't needed, you are including it from /static, not /static/dist.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#issuecomment-597072961
 
 
   Tests are failing

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#issuecomment-610136821
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=h1) Report
   > Merging [#7653](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0dafdd0b9d635b4513b1413007337b19c3d96b17&el=desc) will **decrease** coverage by `0.01%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7653/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7653      +/-   ##
   ==========================================
   - Coverage   88.31%   88.30%   -0.02%     
   ==========================================
     Files         935      935              
     Lines       45170    45173       +3     
   ==========================================
   - Hits        39892    39888       -4     
   - Misses       5278     5285       +7     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/www/blueprints.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvYmx1ZXByaW50cy5weQ==) | `100.00% <100.00%> (ø)` | |
   | [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `70.45% <0.00%> (-8.34%)` | :arrow_down: |
   | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.94% <0.00%> (-0.76%)` | :arrow_down: |
   | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.81% <0.00%> (+0.43%)` | :arrow_up: |
   | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.64% <0.00%> (+1.11%)` | :arrow_up: |
   | [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `94.36% <0.00%> (+1.40%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=footer). Last update [0dafdd0...0bdd464](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#discussion_r404730936
 
 

 ##########
 File path: airflow/www/templates/airflow/dag.html
 ##########
 @@ -21,6 +21,10 @@
 
 {% block page_title %}{{ dag.dag_id }} - Airflow{% endblock %}
 
+{% block head_meta %}
+  <meta name="robots" content="noindex" />
 
 Review comment:
   !! This should call `{{ super() }}`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#issuecomment-610136821
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=h1) Report
   > Merging [#7653](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0dafdd0b9d635b4513b1413007337b19c3d96b17&el=desc) will **decrease** coverage by `0.01%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7653/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7653      +/-   ##
   ==========================================
   - Coverage   88.31%   88.30%   -0.02%     
   ==========================================
     Files         935      935              
     Lines       45170    45173       +3     
   ==========================================
   - Hits        39892    39888       -4     
   - Misses       5278     5285       +7     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/www/blueprints.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvYmx1ZXByaW50cy5weQ==) | `100.00% <100.00%> (ø)` | |
   | [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `70.45% <0.00%> (-8.34%)` | :arrow_down: |
   | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.94% <0.00%> (-0.76%)` | :arrow_down: |
   | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.81% <0.00%> (+0.43%)` | :arrow_up: |
   | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.64% <0.00%> (+1.11%)` | :arrow_up: |
   | [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `94.36% <0.00%> (+1.40%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=footer). Last update [0dafdd0...0bdd464](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#discussion_r404731378
 
 

 ##########
 File path: tests/www/test_views.py
 ##########
 @@ -319,6 +319,11 @@ def test_not_found(self):
         resp = self.client.get('/', follow_redirects=True)
         self.assertEqual(resp.status_code, 404)
 
+    def test_robots_txt(self):
+        resp = self.client.get('/test/robots.txt', follow_redirects=True)
 
 Review comment:
   Does /test/robots.txt do anything? I thought this was  only read if it lives at /robots.txt?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#issuecomment-610136821
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=h1) Report
   > Merging [#7653](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0dafdd0b9d635b4513b1413007337b19c3d96b17&el=desc) will **decrease** coverage by `0.01%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7653/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7653      +/-   ##
   ==========================================
   - Coverage   88.31%   88.30%   -0.02%     
   ==========================================
     Files         935      935              
     Lines       45170    45173       +3     
   ==========================================
   - Hits        39892    39888       -4     
   - Misses       5278     5285       +7     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/www/blueprints.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvYmx1ZXByaW50cy5weQ==) | `100.00% <100.00%> (ø)` | |
   | [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `70.45% <0.00%> (-8.34%)` | :arrow_down: |
   | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.94% <0.00%> (-0.76%)` | :arrow_down: |
   | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.81% <0.00%> (+0.43%)` | :arrow_up: |
   | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.64% <0.00%> (+1.11%)` | :arrow_up: |
   | [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `94.36% <0.00%> (+1.40%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=footer). Last update [0dafdd0...0bdd464](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] retornam commented on issue #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
retornam commented on issue #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#issuecomment-613670159
 
 
   No I think I have enough based on the review, unless you have more to add. Thanks 
   
   > On Apr 14, 2020, at 13:06, Kaxil Naik <no...@github.com> wrote:
   > 
   > 
   > Hey @retornam Do you need any more info to create a new PR to solve the same issue?
   > 
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub, or unsubscribe.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#discussion_r405104784
 
 

 ##########
 File path: tests/www/test_views.py
 ##########
 @@ -319,6 +319,11 @@ def test_not_found(self):
         resp = self.client.get('/', follow_redirects=True)
         self.assertEqual(resp.status_code, 404)
 
+    def test_robots_txt(self):
+        resp = self.client.get('/test/robots.txt', follow_redirects=True)
 
 Review comment:
   But serving /test/robots.txt won't do anything - nothing will ever request it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#discussion_r405106036
 
 

 ##########
 File path: airflow/www/templates/airflow/dag.html
 ##########
 @@ -21,6 +21,10 @@
 
 {% block page_title %}{{ dag.dag_id }} - Airflow{% endblock %}
 
+{% block head_meta %}
+  <meta name="robots" content="noindex" />
 
 Review comment:
   Look at the next block 4 lines down

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil merged pull request #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#issuecomment-610136821
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=h1) Report
   > Merging [#7653](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0dafdd0b9d635b4513b1413007337b19c3d96b17&el=desc) will **decrease** coverage by `0.01%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7653/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7653      +/-   ##
   ==========================================
   - Coverage   88.31%   88.30%   -0.02%     
   ==========================================
     Files         935      935              
     Lines       45170    45173       +3     
   ==========================================
   - Hits        39892    39888       -4     
   - Misses       5278     5285       +7     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/www/blueprints.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvYmx1ZXByaW50cy5weQ==) | `100.00% <100.00%> (ø)` | |
   | [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `70.45% <0.00%> (-8.34%)` | :arrow_down: |
   | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.94% <0.00%> (-0.76%)` | :arrow_down: |
   | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.81% <0.00%> (+0.43%)` | :arrow_up: |
   | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.64% <0.00%> (+1.11%)` | :arrow_up: |
   | [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `94.36% <0.00%> (+1.40%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=footer). Last update [0dafdd0...0bdd464](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ad-m commented on issue #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
ad-m commented on issue #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#issuecomment-598986102
 
 
   @retornam , could you make tests pass and ask GH to assign CVE?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] retornam commented on a change in pull request #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
retornam commented on a change in pull request #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#discussion_r405097960
 
 

 ##########
 File path: tests/www/test_views.py
 ##########
 @@ -319,6 +319,11 @@ def test_not_found(self):
         resp = self.client.get('/', follow_redirects=True)
         self.assertEqual(resp.status_code, 404)
 
+    def test_robots_txt(self):
+        resp = self.client.get('/test/robots.txt', follow_redirects=True)
 
 Review comment:
   @ashb we set the default url in the config for the test to include `/test/`
   https://github.com/apache/airflow/blob/master/tests/www/test_views.py#L303
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] retornam commented on a change in pull request #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
retornam commented on a change in pull request #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#discussion_r405098671
 
 

 ##########
 File path: airflow/www/templates/airflow/dag.html
 ##########
 @@ -21,6 +21,10 @@
 
 {% block page_title %}{{ dag.dag_id }} - Airflow{% endblock %}
 
+{% block head_meta %}
+  <meta name="robots" content="noindex" />
 
 Review comment:
   @ashb  sorry can you be a bit clearer / detailed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#issuecomment-610329723
 
 
   @ad-m Lack of robots.txt is not CVE worthy. All it does is stop indexing -- if you are relying or expecting that to provide any protection you are in for a rude shock.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#issuecomment-610136821
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=h1) Report
   > Merging [#7653](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0dafdd0b9d635b4513b1413007337b19c3d96b17&el=desc) will **decrease** coverage by `0.01%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7653/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7653      +/-   ##
   ==========================================
   - Coverage   88.31%   88.30%   -0.02%     
   ==========================================
     Files         935      935              
     Lines       45170    45173       +3     
   ==========================================
   - Hits        39892    39888       -4     
   - Misses       5278     5285       +7     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/www/blueprints.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvYmx1ZXByaW50cy5weQ==) | `100.00% <100.00%> (ø)` | |
   | [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `70.45% <0.00%> (-8.34%)` | :arrow_down: |
   | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.94% <0.00%> (-0.76%)` | :arrow_down: |
   | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.81% <0.00%> (+0.43%)` | :arrow_up: |
   | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.64% <0.00%> (+1.11%)` | :arrow_up: |
   | [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `94.36% <0.00%> (+1.40%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=footer). Last update [0dafdd0...0bdd464](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#issuecomment-613654409
 
 
   Hey @retornam Do you need any more info to create a new PR to solve the same issue?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#issuecomment-610136821
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=h1) Report
   > Merging [#7653](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0dafdd0b9d635b4513b1413007337b19c3d96b17&el=desc) will **decrease** coverage by `55.05%`.
   > The diff coverage is `75.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7653/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master    #7653       +/-   ##
   ===========================================
   - Coverage   88.31%   33.26%   -55.06%     
   ===========================================
     Files         935      935               
     Lines       45170    45173        +3     
   ===========================================
   - Hits        39892    15025    -24867     
   - Misses       5278    30148    +24870     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/www/blueprints.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvYmx1ZXByaW50cy5weQ==) | `71.42% <75.00%> (-28.58%)` | :arrow_down: |
   | [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [806 more](https://codecov.io/gh/apache/airflow/pull/7653/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=footer). Last update [0dafdd0...e429583](https://codecov.io/gh/apache/airflow/pull/7653?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #7653: [AIRFLOW-6914] Add a default robots.txt

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7653: [AIRFLOW-6914] Add a default robots.txt
URL: https://github.com/apache/airflow/pull/7653#issuecomment-610348247
 
 
   (We've reverted this PR as it's not quite right yet -- we still want the feature though.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services