You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vincbeck (via GitHub)" <gi...@apache.org> on 2023/04/03 16:30:53 UTC
[GitHub] [airflow-site] vincbeck opened a new pull request, #761: Update robots.txt to crawl only latest version of documentation
vincbeck opened a new pull request, #761:
URL: https://github.com/apache/airflow-site/pull/761
See issue [#29283](https://github.com/apache/airflow/issues/29283).
The current robots.txt allows all pages to be indexed by search engines. As a a consequence, some old version of documentation ends up being indexed which can be confusing for users. They end up reading old version of documentation thinking it is the latest one. An example of user getting confused [here](https://apache-airflow.slack.com/archives/CSS36QQS1/p1680524708746749).
Which this change, all pages under `/docs/apache-airflow/` but `/docs/apache-airflow/stable/` **will stop being crawled by search engines**. !! Please confirm this is fine since this change is quite impactful !!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1507443433
> Followed 1. and 2. only. The script docs/build.sh does not exist anymore in Airflow repository so I ran these commands instead to build the doc and visualize them:
Look what breeze does in the `developer_commands.py` for build_docs:
```
cmd = [
"docker",
"run",
"-t",
*extra_docker_flags,
"--pull",
"never",
ci_image_name,
"/opt/airflow/scripts/in_container/run_docs_build.sh",
*doc_builder.args_doc_builder,
```
or you can use `--dry-run` when running breeze command and it will show you exactly what it does.
You can enter breeze and run the commnd manually, and install the sphinx theme locally before I believe (possibly needs some modifications of the shell script)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1512106636
Uploaded : https://pypi.org/project/sphinx-airflow-theme/0.0.12/ , Merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] mik-laj commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "mik-laj (via GitHub)" <gi...@apache.org>.
mik-laj commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1496660277
What do you think about using a stable version in canonical links?
```
https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html#using-released-sources
```
This seems common in the world, but often such versions do not have a version number designation in URL.
For example, the latest stable version:
```
https://docs.streamlit.io/library/api-reference/text/st.markdown
```
Old version:
```
https://docs.streamlit.io/1.19.0/library/api-reference/text/st.markdown#stmarkdown
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] krzysztof-kwitt commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "krzysztof-kwitt (via GitHub)" <gi...@apache.org>.
krzysztof-kwitt commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1496654349
We can always use the `stable` version in canonical links.
`https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html`
All old pages should have their equivalents or appropriate redirects in the stable version.
This means that to the following documentation:
```
https://airflow.apache.org/docs/apache-airflow/2.0.0/concepts.html
```
we can add the following canonical link:
```
https://airflow.apache.org/docs/apache-airflow/stable/concepts.html
```
And then, google will be redirected to the following link.
```
https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html
```
For Google, redirects should not be a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] vincbeck commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1496108762
Doing some testing, please ignore for now new commits in this PR (putting the PR in draft in the meantime). I'll let you know once I have something ready.
Meanwhile, what should be the general rule for providers? In other word, which version should be indexed, and which ones should not (for each provider)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk merged pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #761:
URL: https://github.com/apache/airflow-site/pull/761
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] vincbeck commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1507271379
I am still unable to build the docs with the changes I made to the theme. Here are the steps I have done:
- Built and installed theme package locally by following instructions here https://github.com/apache/airflow-site/tree/main/sphinx_airflow_theme
- Updated `sphinx_airflow_theme/sphinx_airflow_theme/layout.html`. You can find these changes in this PR
- Follow instructions here https://github.com/apache/airflow-site/blob/main/CONTRIBUTE.md#instruction
- Followed 1. and 2. only. The script `docs/build.sh` does not exist anymore in Airflow repository so I ran these commands instead to build the doc and visualize them:
- `breeze build-docs --package-filter apache-airflow-providers-amazon`
- ./start_doc_server.sh
It builds the docs successfully but without my changes in `sphinx_airflow_theme/sphinx_airflow_theme/layout.html`. To be honest it makes sense because I do not know how breeze would know it should use my local airflow-site repository as sphinx theme configuration. This is the part I am missing I guess
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1512107133
I **hope** it's going to work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1512065121
Coool! One more thing - can you please update version of the "sphinx-theme" package @vincbeck - then I **think** I should be able to publish it to PyPI (I think I have permissions :D).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1496014626
> Yeah I am happy with that. But we should add the big red banner with warning that this is an old version and provide link for stable version.
Yeah would be nice if someone is brave enough to dive into the depths of our docs building system to do such conditional banner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] mik-laj commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "mik-laj (via GitHub)" <gi...@apache.org>.
mik-laj commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1496655439
We can always use the `stable` version in canonical links.
`https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html`
All old pages should have their equivalents or appropriate redirects in the stable version.
This means that to the following documentation:
```
https://airflow.apache.org/docs/apache-airflow/2.0.0/concepts.html
```
we can add the following canonical link:
```
https://airflow.apache.org/docs/apache-airflow/stable/concepts.html
```
And then, google will be redirected to the following link.
```
https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html
```
For Google, redirects should not be a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1496363194
> * Providers. Hard question. My personal opinion is we should come up with a rule which applies to all providers. Having provider specific rules would be too complex/time spending to manage. That being said I dont have strong opinion on which rule we should have for providers. e.g. Only versions which belong the latest major version? Only versions which belong the 2 latest major versions? WDYT?
Keeping 2 last major versions would be my preference. Reasoning: We usually don't change a lot between major versions. But if we remove an object, we do not want it to be indexed rather quickly - but not immediately. And I doubt we will have such removal in two majore releases one after the other. Usually we do not remove stuff with every major bump, we accumulate it and remove in single "rather big breaking provider release" or at least it happened like that so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] o-nikolas commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1495310526
> if they want to read documentation about a specific version they can select it from the left dropdown
Yeah, for sure, they'll still be able to find the docs, but if the user is knowingly running an older version of Airflow (and with this change older can mean just a single release back)it is helpful to be able to google search for terms that may only exist in the docs for the version which apply to you.
But if it's not possible then I suppose it's moot. I'm interested to hear what others think, because I agree with you that it's quite an impactful change! Perhaps it's worth an email on the dev list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] mik-laj commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "mik-laj (via GitHub)" <gi...@apache.org>.
mik-laj commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1501110687
You're absolutely right.
```
https://airflow.apache.org/docs/apache-airflow/stable/index.html
```
is the canonical link for Airflow documentation.
```
https://airflow.apache.org/docs/apache-airflow/2.5.3/index.html
```
is NOT the canonical link. It is an archival version.
I think you will have to make changes in two places:
- in the template so that newly generated docs contain a `link` tag. See: https://github.com/apache/airflow-site/tree/main/sphinx_airflow_theme
- archival docs: https://github.com/apache/airflow-site/tree/main/docs-archive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] vincbeck commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1499290674
Basically the canonical link for each package would be the stable one right?
Example:
- https://airflow.apache.org/docs/apache-airflow/stable/index.html is the canonical link for Airflow documentation. https://airflow.apache.org/docs/apache-airflow/2.5.3/index.html is not the canonical link even though, today, it goes to the same page
Is it was you are suggesting?
I do not know much about documentation generation in Airflow and the work required in order to do so but it seems doable. For each package, we just need to add a `<link tel="canonical"` which specifies the stable link as the canonical one. Am I wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1505958273
This was always a bit of mistery to me especially when you make change to a theme. But:
* Building and installing theme package locally https://github.com/apache/airflow-site/tree/main/sphinx_airflow_theme
* Instruction to build the docs https://github.com/apache/airflow-site/blob/main/CONTRIBUTE.md#instruction
I think some of the docs/versions are a bit outdated and might require some trickery to install on newer systems though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1508627822
> So I think the issue you had was basically that the original script is moved :)
I hope the result of this exercise will also be updating the docs :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] vincbeck commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1496581427
Interesting. I did know that. It looks like a very good fit for our problem indeed. The doc actually mention "Don't use the robots.txt file for canonicalization purposes.". And by reading the doc and definition of canonicalization, it looks like our problem here is very canonicalization. Any other opinion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1495805244
I think I'd go half-way. Remove All 2.0, 2.1, 2.2, 1.10 docs from the index. That would likely be a much better choice. We already have somewhat soft deprecation mechanism via providers. It's not "you should upgrade" but "if you don't, latest provider releases will not be installable for you". In May we bump it to 2.4, so we can add 2.3 to the exclusions too.
While this is not a "strict" deprecation mechanism, I think it's as good as can be to push people in the direction of upgrade. I think not indexing those docs is a good reflection of our more or less gentle suggestion to upgrade and reflecting that in the index is a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] mik-laj commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "mik-laj (via GitHub)" <gi...@apache.org>.
mik-laj commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1496558935
What do you think about adding <link tel="canonical" to suggest Google best version of site? It is preferred way by Google to consolidate duplicate pages.
https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls?hl=en
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1496015676
(Hint): I am not brave enough :scream:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] vincbeck commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1512069059
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] vincbeck commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1511561168
@potiuk I am not able to locate this `.whl` file. Any hint on where I can find it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] vincbeck commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1496345696
I found a way to fetch all versions from providers and Airflow. I am using the file `packages-metadata.json`. That way we can apply some deterministic rules to decide whether a given version should be indexed. The current implementation disallows all versions but the last one. **I understand this is not what we want and just did that for example**. Here is the
[robots.txt](https://github.com/apache/airflow-site/files/11151148/robots.txt) file generated with this implementation.
I would like to use this PR to discuss and decide on which condition a given version (for Airflow and providers) should be indexed by search engines.
- Airflow: @potiuk suggested to have an upper limit (currently `2.3` and will potentially bump up to `2.4` in May). Everything **below** this upper limit would NOT be indexed by search engines. This upper limit is the minimum Airflow version needed to install latest provider releases
- Providers. Hard question. My personal opinion is we should come up with a rule which applies to all providers. Having provider specific rules would be too complex/time spending to manage. That being said I dont have strong opinion on which rule we should have for providers. e.g. Only versions which belong the latest major version? Only versions which belong the 2 latest major versions? WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1496593374
That's an interesting idea @mik-laj :) . It's not exactly why canonical was created for, but it might do it in our case. However this is significantly more complex to implement because not only we would have to post-process historical pages and add "canonical" link, but we would have to continuously update old pages as they get older - note that "stable" is a symbolic link to the last version, so in effect the stable links would refer to themselves,
This on it's own is not a bad practice - it seems to be even recommended https://www.searchenginejournal.com/google-self-referencing-canonicals-are-not-critical/312619/ but then we either post-process the docs when some new version is released or we are limiting ourselves strictly to only indexing the very last patchleve version for airflow or providers.
IMHO - If we choose to only index the last patchlevel - then using canonical makes sense. If we choose another scheme, this is a lot of hassle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] vincbeck commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1503584113
Alright, I'll do that then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1511624718
> @potiuk I am not able to locate this `.whl` file. Any hint on where I can find it?
You should build it - I think one of the steps (with site.sh script does it) https://github.com/apache/airflow-site/blob/main/sphinx_airflow_theme/README.md (and if not that's I guess pretty standard python package that can be built with `python setup.py` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] vincbeck commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1505913950
I updated the PR to include the canonical link in newly generated doc. HOWEVER, I could not find any way to really test the doc generation with this new layout. The only way I could find was using `demo` but I'd like to generate the actual docs with this new layout. Anyone knows how can I do that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1503376025
My take:
* As long as we want to restrict indexing to "only" stable. Canonical is good idea.
* We indeed need to manually update the existing link in all the past html pages :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] vincbeck commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1505789893
I'll use this PR to add the canonical link to newly generated docs. Of course I'll revert the changes I mades to robots.txt as well. I'll create another PR to update archived docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1507447346
BTW., The `--dry-run` command shows copy-pasteable command that you can run manually to see what breeze does. - then you can apply any modification to those or try to mimic it manually.
This is the actual command when I run it on my machine:
```
AIRFLOW_CI_IMAGE="ghcr.io/apache/airflow/main/ci/python3.7" \
AIRFLOW_CI_IMAGE_WITH_TAG="ghcr.io/apache/airflow/main/ci/python3.7" \
AIRFLOW_CONSTRAINTS_MODE="constraints-source-providers" \
AIRFLOW_CONSTRAINTS_REFERENCE="constraints-main" \
AIRFLOW_ENABLE_AIP_44="true" \
AIRFLOW_ENABLE_AIP_52="true" \
AIRFLOW_EXTRAS="devel_ci" \
AIRFLOW_IMAGE_KUBERNETES="" \
AIRFLOW_PROD_IMAGE="ghcr.io/apache/airflow/main/ci/python3.7" \
AIRFLOW_SOURCES="" \
AIRFLOW_VERSION="" \
ANSWER="" \
BACKEND="" \
BASE_BRANCH="" \
BREEZE="true" \
BREEZE_INIT_COMMAND="" \
CI="false" \
CI_BUILD_ID="0" \
CI_EVENT_TYPE="pull_request" \
CI_JOB_ID="0" \
CI_TARGET_BRANCH="main" \
CI_TARGET_REPO="apache/airflow" \
COLLECT_ONLY="false" \
COMMIT_SHA="COMMIT_SHA_NOT_FOUND" \
COMPOSE_FILE="" \
DB_RESET="" \
DEFAULT_BRANCH="main" \
DEFAULT_CONSTRAINTS_BRANCH="constraints-main" \
DEV_MODE="" \
ENABLED_SYSTEMS="" \
ENABLE_TEST_COVERAGE="false" \
FLOWER_HOST_PORT="25555" \
GITHUB_ACTIONS="false" \
HOST_GROUP_ID="" \
HOST_OS="linux" \
HOST_USER_ID="" \
INIT_SCRIPT_FILE="init.sh" \
INSTALL_AIRFLOW_VERSION="" \
INSTALL_PACKAGES_FROM_CONTEXT="false" \
INSTALL_PROVIDERS_FROM_SOURCES="false" \
ISSUE_ID="" \
LOAD_DEFAULT_CONNECTIONS="" \
LOAD_EXAMPLES="" \
MSSQL_DATA_VOLUME="" \
MSSQL_HOST_PORT="21433" \
MSSQL_VERSION="" \
MYSQL_HOST_PORT="23306" \
MYSQL_VERSION="" \
NUM_RUNS="" \
PACKAGE_FORMAT="" \
POSTGRES_HOST_PORT="25433" \
POSTGRES_VERSION="" \
PYTHONDONTWRITEBYTECODE="true" \
PYTHON_MAJOR_MINOR_VERSION="3.7" \
REDIS_HOST_PORT="26379" \
REMOVE_ARM_PACKAGES="false" \
RUN_SYSTEM_TESTS="false" \
RUN_TESTS="false" \
SKIP_CONSTRAINTS="" \
SKIP_ENVIRONMENT_INITIALIZATION="" \
SKIP_PROVIDER_TESTS="" \
SKIP_SSH_SETUP="false" \
SQLITE_URL="" \
SSH_PORT="12322" \
START_AIRFLOW="" \
SUSPENDED_PROVIDERS_FOLDERS="" \
TEST_TIMEOUT="60" \
TEST_TYPE="" \
UPGRADE_BOTO="" \
UPGRADE_TO_NEWER_DEPENDENCIES="false" \
USE_AIRFLOW_VERSION="" \
USE_PACKAGES_FROM_DIST="" \
VERBOSE="false" \
VERBOSE_COMMANDS="false" \
VERSION_SUFFIX_FOR_PYPI="" \
WEBSERVER_HOST_PORT="28080" \
WHEEL_VERSION="0.36.2" \
docker run -t --mount type=bind,src=/home/jarek/code/airflow/.bash_aliases,dst=/root/.bash_aliases --mount type=bind,src=/home/jarek/code/airflow/.bash_history,dst=/root/.bash_history --mount type=bind,src=/home/jarek/code/airflow/.build,dst=/opt/airflow/.build --mount type=bind,src=/home/jarek/code/airflow/.coveragerc,dst=/opt/airflow/.coveragerc --mount type=bind,src=/home/jarek/code/airflow/.dockerignore,dst=/opt/airflow/.dockerignore --mount type=bind,src=/home/jarek/code/airflow/.github,dst=/opt/airflow/.github --mount type=bind,src=/home/jarek/code/airflow/.inputrc,dst=/root/.inputrc --mount type=bind,src=/home/jarek/code/airflow/.rat-excludes,dst=/opt/airflow/.rat-excludes --mount type=bind,src=/home/jarek/code/airflow/BREEZE.rst,dst=/opt/airflow/BREEZE.rst --mount type=bind,src=/home/jarek/code/airflow/LICENSE,dst=/opt/airflow/LICENSE --mount type=bind,src=/home/jarek/code/airflow/MANIFEST.in,dst=/opt/airflow/MANIFEST.in --mount type=bind,src=/home/jarek/code/airflow/NOTIC
E,dst=/opt/airflow/NOTICE --mount type=bind,src=/home/jarek/code/airflow/RELEASE_NOTES.rst,dst=/opt/airflow/RELEASE_NOTES.rst --mount type=bind,src=/home/jarek/code/airflow/airflow,dst=/opt/airflow/airflow --mount type=bind,src=/home/jarek/code/airflow/constraints,dst=/opt/airflow/constraints --mount type=bind,src=/home/jarek/code/airflow/provider_packages,dst=/opt/airflow/provider_packages --mount type=bind,src=/home/jarek/code/airflow/dags,dst=/opt/airflow/dags --mount type=bind,src=/home/jarek/code/airflow/dev,dst=/opt/airflow/dev --mount type=bind,src=/home/jarek/code/airflow/docs,dst=/opt/airflow/docs --mount type=bind,src=/home/jarek/code/airflow/generated,dst=/opt/airflow/generated --mount type=bind,src=/home/jarek/code/airflow/hooks,dst=/opt/airflow/hooks --mount type=bind,src=/home/jarek/code/airflow/images,dst=/opt/airflow/images --mount type=bind,src=/home/jarek/code/airflow/logs,dst=/root/airflow/logs --mount type=bind,src=/home/jarek/code/airflow/pyproject.toml,dst=/opt
/airflow/pyproject.toml --mount type=bind,src=/home/jarek/code/airflow/scripts,dst=/opt/airflow/scripts --mount type=bind,src=/home/jarek/code/airflow/scripts/docker/entrypoint_ci.sh,dst=/entrypoint --mount type=bind,src=/home/jarek/code/airflow/setup.cfg,dst=/opt/airflow/setup.cfg --mount type=bind,src=/home/jarek/code/airflow/setup.py,dst=/opt/airflow/setup.py --mount type=bind,src=/home/jarek/code/airflow/tests,dst=/opt/airflow/tests --mount type=bind,src=/home/jarek/code/airflow/kubernetes_tests,dst=/opt/airflow/kubernetes_tests --mount type=bind,src=/home/jarek/code/airflow/docker_tests,dst=/opt/airflow/docker_tests --mount type=bind,src=/home/jarek/code/airflow/chart,dst=/opt/airflow/chart --mount type=bind,src=/home/jarek/code/airflow/files,dst=/files --mount type=bind,src=/home/jarek/code/airflow/dist,dst=/dist --rm --env-file /home/jarek/code/airflow/scripts/ci/docker-compose/_docker.env --pull never ghcr.io/apache/airflow/main/ci/python3.7 /opt/airflow/scripts/in_container
/run_docs_build.sh
```
When you remove the last scripts, you will enter breeze env and you will be able to manually run the script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] vincbeck commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1495987528
> > I think I'd go half-way. Remove All 2.0, 2.1, 2.2, 1.10 docs from the index. That would likely be a much better choice. We already have somewhat soft deprecation mechanism via providers. It's not "you should upgrade" but "if you don't, latest provider releases will not be installable for you". In May we bump it to 2.4, so we can add 2.3 to the exclusions too.
>
> Yeah I am happy with that. But we should add the big red banner with warning that this is an old version and provide link for stable version.
>
> Example with SQLAlchemy: (https://docs.sqlalchemy.org/en/14/intro.html#installation)
>
> <img alt="image" width="387" src="https://user-images.githubusercontent.com/8811558/229796562-b2f45721-2e34-42f3-b2b1-b7d1c28f1bf3.png">
>
> Example with [Astro SDK](https://astro-sdk-python.readthedocs.io/en/1.5.2/) (that uses ReadTheDocs): <img alt="image" width="1067" src="https://user-images.githubusercontent.com/8811558/229796841-5452a2b7-e864-48b2-bb5f-a62647667558.png">
Agree with this. Though, I was thinking splitting the change in two PRs, this one exclusively about the robots.txt change and then I (or someone else?) will create another one to add the banner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] vincbeck commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1494979059
As far as I know it is only enable/disable. See documentation [here](https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt). And my personal opinion is I rather not index old docs so users are always redirected to the most up to date docs and if they want to read documentation about a specific version they can select it from the left dropdown
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] o-nikolas commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1494971973
Is there a way to provide a weight to the stable docs and still keep the older docs indexed (but they will just be less likely to show up unless searching for terms only contained in older docs)? Or is it only enable/disable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] kaxil commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1495916893
> I think I'd go half-way. Remove All 2.0, 2.1, 2.2, 1.10 docs from the index. That would likely be a much better choice. We already have somewhat soft deprecation mechanism via providers. It's not "you should upgrade" but "if you don't, latest provider releases will not be installable for you". In May we bump it to 2.4, so we can add 2.3 to the exclusions too.
Yeah I am happy with that. But we should add the big red banner with warning that this is an old version and provide link for stable version.
Example with SQLAlchemy: (https://docs.sqlalchemy.org/en/14/intro.html#installation)
<img width="387" alt="image" src="https://user-images.githubusercontent.com/8811558/229796562-b2f45721-2e34-42f3-b2b1-b7d1c28f1bf3.png">
Example with Astro SDK (that uses ReadTheDocs):
<img width="1067" alt="image" src="https://user-images.githubusercontent.com/8811558/229796841-5452a2b7-e864-48b2-bb5f-a62647667558.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1508624074
BTW. @vincbeck I think it should be **just** enough to copy the theme to /files folder, enter install the theme from there `pip install /files/your.whl` and then run the " /opt/airflow/scripts/in_container/run_docs_build.sh" script with the right args .... You should not even need to run the `--dry-run` or copy&paste anything.
The `run_docs_build.sh` has `--help` and it will print you the options (they are basically the same as in breeze command).
So I think the issu you had was basically that the original script is moved :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] vincbeck commented on pull request #761: Update robots.txt to crawl only latest version of documentation
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1512052016
I could finally tested my changes. It was correct. I updated the doc and added a new section to explain the steps I followed so if anyone needs to do the same thing, it should be easy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org