You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "bitsondatadev (via GitHub)" <gi...@apache.org> on 2023/10/25 11:43:19 UTC
[PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
bitsondatadev opened a new pull request, #8919:
URL: https://github.com/apache/iceberg/pull/8919
Revert back to using the `site` directory which contains the monorepo of the site and versioned docs. Also move back to using the asf infra standards including the [asf-site](https://github.com/apache/iceberg/tree/asf-site) branch.
I gleaned info from [this PR](https://github.com/apache/iceberg/pull/4062/files), and gleaning a bit off of [this deploy script](https://github.com/apache/iceberg-docs/blob/main/.github/workflows/deploy.yml).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1386414038
##########
site/dev/common.sh:
##########
@@ -0,0 +1,125 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+mv_nightly_up () {
+ mv docs/docs/nightly/ docs/
+}
+
+mv_nightly_down () {
+ mv docs/nightly/ docs/docs/
+}
+
+uninstall_deps () {
+ if test -d site_env; then
+ deactivate
+ rm -r site_env
+ fi
+}
+
+install_deps () {
+ uninstall_deps
+ python -m venv site_env
+ source site_env/bin/activate
+ pip install -r requirements.txt
+}
+
+assert_not_empty () {
+ if [ -z "$1" ]
+ then
+ echo "No argument supplied"
+ exit 1
+ fi
+}
+
+get_latest_version () {
+ basename $(ls -d docs/docs/*/ | sort -V | tail -1)
+}
+
+update_version () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ sed -i '' -E "s/(^site\_name:[[:space:]]+docs\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+ sed -i '' -E "s/(^[[:space:]]*-[[:space:]]+Javadoc:.*\/javadoc\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+
+ sed -i '' -E "s/ \- latest: '\!include docs\/docs\/latest\/mkdocs\.yml'/a \- ${ICEBERG_VERSION}: '\!include docs\/docs\/${ICEBERG_VERSION}\/mkdocs\.yml/" ../../mkdocs.yml
+}
+
+create_latest () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ rm -rf docs/docs/latest/
+ mkdir docs/docs/latest/
+
+ ln -s "../${ICEBERG_VERSION}/docs" docs/docs/latest/docs
+ cp "docs/docs/${ICEBERG_VERSION}/mkdocs.yml" docs/docs/latest/
+
+ cd docs/docs/
+ update_version "latest"
+ cd -
+}
+
+
+# https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/#search-exclusion
+search_exclude_versioned_docs () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ cd "${ICEBERG_VERSION}/docs/"
+
+ python3 -c "import os
+for f in filter(lambda x: x.endswith('.md'), os.listdir()): lines = open(f).readlines(); open(f, 'w').writelines(lines[:2] + ['search:\n', ' exclude: true\n'] + lines[2:]);"
+
+ cd -
+}
+
+pull_versioned_docs () {
+ mv_nightly_up
+
+ git worktree add docs/docs docs
+ git worktree add docs/javadoc javadoc
Review Comment:
Should be good now!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1443361277
##########
site/docs/community.md:
##########
@@ -40,13 +40,13 @@ Issues are tracked in GitHub:
## Slack
-We use the [Apache Iceberg workspace](https://apache-iceberg.slack.com/) on Slack. To be invited, follow [this invite link](https://join.slack.com/t/apache-iceberg/shared_invite/zt-1znkcg5zm-7_FE~pcox347XwZE3GNfPg).
+We use the [Apache Iceberg workspace](https://apache-iceberg.slack.com/) on Slack. To be invited, follow [this invite link](https://join.slack.com/t/apache-iceberg/shared_invite/zt-287g3akar-K9Oe_En5j1UL7Y_Ikpai3A).
Review Comment:
wait yeah this is the latest one haha and that's what's currently in there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1371606620
##########
site/README.md:
##########
@@ -65,7 +65,7 @@ The non-versioned site pages are all the `/site/docs/.*md` files and the docs ar
> This build process is currently missing older versions and the javadoc branches.
> Until these branches are merged, these steps will not work.
-All previously versioned docs will be committed in `docs-<version>` branches and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree will pull these versions in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs in a single tag.
+The Iceberg versioned docs are committed in the `docs` branch and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree pulls these version directories in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs.
Review Comment:
```suggestion
The Iceberg versioned docs are committed in the `docs` branch and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree pulls these version directories in by following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs.
```
##########
site/README.md:
##########
@@ -125,17 +124,16 @@ mkdocs serve
## Release process
Deploying a version of the docs is a two step process:
- 1. ~~Cut a new release from the current branch revision. This creates a new branch `docs-<version>`.~~
+ 1. ~~Release a new version by copying the current nightly directory to a new version directory in the `docs` branch.~~
Review Comment:
what does `nightly directory` refer to?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1389531071
##########
site/mkdocs.yml:
##########
@@ -23,33 +23,37 @@ theme:
logo: assets/images/iceberg-logo-icon.png
favicon: assets/images/favicon-96x96.png
features:
+ - content.tabs.link
+ - content.code.copy
+ - content.code.annotate
- navigation.tabs
- navigation.path
- navigation.top
- navigation.tracking
- - toc.follow
+ - offline
- search.suggest
- search.highlight
- - content.tabs.link
- - content.code.copy
- - content.code.annotate
+ - toc.follow
plugins:
- search
- macros:
include_yaml:
- variables.yml
- monorepo
+ - offline:
+ enabled: !ENV [OFFLINE, false]
nav:
- Quickstart:
- Spark: spark-quickstart.md
- Hive: hive-quickstart.md
- Docs:
- nightly: '!include docs/docs/nightly/mkdocs.yml'
-# - latest: '!include docs/docs/latest/mkdocs.yml'
-# - 1.3.1: '!include docs/docs/1.3.1/mkdocs.yml'
-# - 1.3.0: '!include docs/docs/1.3.0/mkdocs.yml'
+ - latest: '!include docs/docs/latest/mkdocs.yml'
Review Comment:
I've gotten mixed signals depending who I've spoken to about meta refresh for redirects.
According to most of the stuff that comes up it's [not as big of a deal](https://webmasters.stackexchange.com/a/141240) as some made it out to be. So we can likely change `latest` to being a refresh page, but I'm curious to see how mkdocs renders that.
We should ideally have latest get ranked by Google and we may just want to be cognizant if we get a bunch of random versions or if latest gets ranked higher.
It's a bit of lore and black box when it comes to ensuring a certain version gets indexed and ranked by Google. No real way to control it so this was my feeble attempt.
<https://developers.google.com/search/docs/crawling-indexing/301-redirects#metarefresh>
Based on their docs, I think we should be fine if we update all the pages to set the [`site_url`](https://www.mkdocs.org/user-guide/configuration/#site_url) which will always point to latest.
Also, we'll likely want to add an [outdated version warning](https://squidfunk.github.io/mkdocs-material/setup/setting-up-versioning/#version-warning) which will link to the "latest" equivalent of the page if available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1386208436
##########
site/dev/common.sh:
##########
@@ -0,0 +1,125 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+mv_nightly_up () {
+ mv docs/docs/nightly/ docs/
+}
+
+mv_nightly_down () {
+ mv docs/nightly/ docs/docs/
+}
+
+uninstall_deps () {
+ if test -f .env; then
+ deactivate
+ rm -r .env
+ fi
+}
+
+install_deps () {
+ uninstall_deps
+ python -m venv .env
+ source env/bin/activate
+ pip install -r requirements.txt
+}
+
+assert_not_empty () {
+ if [ -z "$1" ]
+ then
+ echo "No argument supplied"
+ exit 1
+ fi
+}
+
+get_latest_version () {
+ basename $(ls -d docs/docs/*/ | sort -V | tail -1)
+}
+
+update_version () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ sed -i '' -E "s/(^site\_name:[[:space:]]+docs\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+ sed -i '' -E "s/(^[[:space:]]*-[[:space:]]+Javadoc:.*\/javadoc\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+
+ sed -i '' -E "s/(^ \- latest: '\!include docs\/docs\/latest\/mkdocs\.yml'/a \- ${ICEBERG_VERSION}: '\!include docs\/docs\/${ICEBERG_VERSION}\/mkdocs\.yml/" ../../mkdocs.yml
+}
+
+create_latest () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ rm -rf docs/docs/latest/
+ mkdir docs/docs/latest/
+
+ ln -s "../${ICEBERG_VERSION}/docs" docs/docs/latest/docs
+ cp "docs/docs/${ICEBERG_VERSION}/mkdocs.yml" docs/docs/latest/
+
+ cd docs/docs/
+ update_version "latest"
+ cd -
+}
+
+
+# https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/#search-exclusion
+search_exclude_versioned_docs () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ cd "${ICEBERG_VERSION}/docs/"
+
+ python3 -c "import os
Review Comment:
yeesh, yeah...I think best practice is python3 no?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1389531071
##########
site/mkdocs.yml:
##########
@@ -23,33 +23,37 @@ theme:
logo: assets/images/iceberg-logo-icon.png
favicon: assets/images/favicon-96x96.png
features:
+ - content.tabs.link
+ - content.code.copy
+ - content.code.annotate
- navigation.tabs
- navigation.path
- navigation.top
- navigation.tracking
- - toc.follow
+ - offline
- search.suggest
- search.highlight
- - content.tabs.link
- - content.code.copy
- - content.code.annotate
+ - toc.follow
plugins:
- search
- macros:
include_yaml:
- variables.yml
- monorepo
+ - offline:
+ enabled: !ENV [OFFLINE, false]
nav:
- Quickstart:
- Spark: spark-quickstart.md
- Hive: hive-quickstart.md
- Docs:
- nightly: '!include docs/docs/nightly/mkdocs.yml'
-# - latest: '!include docs/docs/latest/mkdocs.yml'
-# - 1.3.1: '!include docs/docs/1.3.1/mkdocs.yml'
-# - 1.3.0: '!include docs/docs/1.3.0/mkdocs.yml'
+ - latest: '!include docs/docs/latest/mkdocs.yml'
Review Comment:
I've gotten mixed signals depending who I've spoken to about meta refresh for redirects.
According to most of the stuff that comes up it's [not as big of a deal](https://webmasters.stackexchange.com/a/141240) as some made it out to be. So we can likely change `latest` to being a refresh page, but I'm curious to see how mkdocs renders that.
We should ideally have latest get ranked by Google and we may just want to be cognizant if we get a bunch of random versions or if latest gets ranked higher.
It's a bit of lore and black box when it comes to ensuring a certain version gets indexed and ranked by Google. No real way to control it so this was my feeble attempt.
<https://developers.google.com/search/docs/crawling-indexing/301-redirects#metarefresh>
Based on their docs, I think we should be fine if we update all the pages to set the [`site_url`](https://www.mkdocs.org/user-guide/configuration/#site_url) which will always point to latest.
Also, we'll likely want to add an [outdated version warning](https://squidfunk.github.io/mkdocs-material/setup/setting-up-versioning/#version-warning) which will link to the "latest" equivalent of the page if available, thus sending yet another signal to Google not to rank anything that's not the latest higher.
Okay, all that said, yeah...I think we can remove that and just use a meta refresh and it hopefully won't affect search engine rankings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1391601671
##########
site/dev/common.sh:
##########
@@ -0,0 +1,124 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+REMOTE="iceberg_docs"
+
+create_or_update_docs_remote () {
+ # check if remote exists before adding it
+ git config "remote.${REMOTE}.url" >/dev/null ||
+ git remote add "${REMOTE}" https://github.com/apache/iceberg.git
+
+ git fetch "${REMOTE}"
+}
+
+pull_remote () {
+ local BRANCH="$1"
+ assert_not_empty "${BRANCH}"
+
+ git pull "${REMOTE}" "${BRANCH}"
+}
+
+push_remote () {
+ local BRANCH="$1"
+ assert_not_empty "${BRANCH}"
+
+ git push "${REMOTE}" "${BRANCH}"
+}
+
+install_deps () {
+ pip -q install -r requirements.txt --upgrade
+}
+
+assert_not_empty () {
Review Comment:
not a huge deal but may help to pass through an error message so we know which arg is missing
Echo "Missing an argument: $2 "
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1376383741
##########
site/README.md:
##########
@@ -125,17 +124,16 @@ mkdocs serve
## Release process
Deploying a version of the docs is a two step process:
- 1. ~~Cut a new release from the current branch revision. This creates a new branch `docs-<version>`.~~
+ 1. ~~Release a new version by copying the current nightly directory to a new version directory in the `docs` branch.~~
```
.github/bin/deploy_docs.sh -v 1.4.0
Review Comment:
I know that it is unrelated to this PR, but this file does not seem to exists?
##########
site/README.md:
##########
@@ -125,17 +124,16 @@ mkdocs serve
## Release process
Deploying a version of the docs is a two step process:
- 1. ~~Cut a new release from the current branch revision. This creates a new branch `docs-<version>`.~~
+ 1. ~~Release a new version by copying the current nightly directory to a new version directory in the `docs` branch.~~
```
.github/bin/deploy_docs.sh -v 1.4.0
Review Comment:
I know that it is unrelated to this PR, but this file does not seem to exist?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1378017052
##########
site/README.md:
##########
@@ -65,7 +65,7 @@ The non-versioned site pages are all the `/site/docs/.*md` files and the docs ar
> This build process is currently missing older versions and the javadoc branches.
> Until these branches are merged, these steps will not work.
-All previously versioned docs will be committed in `docs-<version>` branches and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree will pull these versions in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs in a single tag.
+The Iceberg versioned docs are committed in the `docs` branch and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree pulls these version directories in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs.
Review Comment:
Yes, I moved to this assumption so we don't have to make branches per version anymore. I think that will make changes across multiple docs much faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1386194229
##########
site/.gitignore:
##########
@@ -1,13 +1,3 @@
-## Temp remove for first phase
Review Comment:
Yeah, fair enough...I added a few specifics from my previous one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1371601349
##########
.github/workflows/spark-ci.yml:
##########
@@ -141,4 +142,4 @@ jobs:
with:
name: test logs
path: |
- **/build/testlogs
\ No newline at end of file
+ **/build/testlogs
Review Comment:
nit: unnecessary change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1388598336
##########
.github/workflows/site-ci.yml:
##########
@@ -0,0 +1,40 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+name: site-ci
+on:
+ push:
+ branches:
+ - main
+ paths:
+ - site/**
+permissions:
+ contents: write
Review Comment:
Interesting, this wasn't needed for Python: https://github.com/apache/iceberg-python/blob/main/.github/workflows/python-ci-docs.yml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1386152391
##########
site/dev/common.sh:
##########
@@ -0,0 +1,125 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+mv_nightly_up () {
+ mv docs/docs/nightly/ docs/
+}
+
+mv_nightly_down () {
+ mv docs/nightly/ docs/docs/
+}
+
+uninstall_deps () {
+ if test -f .env; then
+ deactivate
+ rm -r .env
+ fi
+}
+
+install_deps () {
+ uninstall_deps
+ python -m venv .env
Review Comment:
This already assumes that you have `virtualenv` installed. Do we actually need a `venv` for this simple setup? We could also use the system `python3`
##########
site/README.md:
##########
@@ -54,18 +54,34 @@ The non-versioned site pages are all the `/site/docs/.*md` files and the docs ar
│ ├── about.md
│ ├── ...
│ └── view-spec.md
-├── README.md
+├── ...
+├── Makefile
├── mkdocs.yml (non-versioned)
-├── requirements.txt
-└── variables.yml
+└── requirements.txt
```
### Building the versioned docs
-> [!IMPORTANT]
-> This build process is currently missing older versions and the javadoc branches.
-> Until these branches are merged, these steps will not work.
+The Iceberg versioned docs are committed in the orphan `docs` branch and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The `docs` branch contains the versioned documenation source files at the root. These versions exist at the `/site/docs/docs/<version>` directory once mounted. The `latest` version, is a soft link to the most recent build version in the `docs` branch. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs that is mounted at `/site/docs/javadoc/<version>`.
-All previously versioned docs will be committed in `docs-<version>` branches and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree will pull these versions in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs in a single tag.
+The docs are built, run, and released using [make](https://www.gnu.org/software/make/manual/make.html). The [Makefile](Makefile) and the [common shell script](dev/common.sh) support the following command:
+
+```
+site > make help
+[build](dev/build.sh): Clean and build the site locally.
+[clean](dev/clean.sh): Clean the local site.
+[deploy](dev/deploy.sh): Clean, build, and deploy the Iceberg docs site.
+help: Show help for each of the Makefile recipes.
+[release](dev/release.sh): Release the current nightly docs as ICEBERG_VERSION (make release ICEBERG_VERSION=<MAJOR.MINOR.PATCH>).
+[serve](dev/serve.sh): Clean, build, and run the site locally.
+```
Review Comment:
The links don't render inside of the code block:
```suggestion
- [build](dev/build.sh): Clean and build the site locally.
- [clean](dev/clean.sh): Clean the local site.
- [deploy](dev/deploy.sh): Clean, build, and deploy the Iceberg docs site.
- [release](dev/release.sh): Release the current nightly docs as ICEBERG_VERSION (make release ICEBERG_VERSION=<MAJOR.MINOR.PATCH>).
- [serve](dev/serve.sh): Clean, build, and run the site locally.
```
##########
site/dev/common.sh:
##########
@@ -0,0 +1,125 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+mv_nightly_up () {
+ mv docs/docs/nightly/ docs/
+}
+
+mv_nightly_down () {
+ mv docs/nightly/ docs/docs/
+}
+
+uninstall_deps () {
+ if test -f .env; then
+ deactivate
+ rm -r .env
+ fi
+}
+
+install_deps () {
+ uninstall_deps
+ python -m venv .env
+ source env/bin/activate
+ pip install -r requirements.txt
+}
+
+assert_not_empty () {
+ if [ -z "$1" ]
+ then
+ echo "No argument supplied"
+ exit 1
+ fi
+}
+
+get_latest_version () {
+ basename $(ls -d docs/docs/*/ | sort -V | tail -1)
+}
+
+update_version () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ sed -i '' -E "s/(^site\_name:[[:space:]]+docs\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+ sed -i '' -E "s/(^[[:space:]]*-[[:space:]]+Javadoc:.*\/javadoc\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+
+ sed -i '' -E "s/(^ \- latest: '\!include docs\/docs\/latest\/mkdocs\.yml'/a \- ${ICEBERG_VERSION}: '\!include docs\/docs\/${ICEBERG_VERSION}\/mkdocs\.yml/" ../../mkdocs.yml
+}
+
+create_latest () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ rm -rf docs/docs/latest/
+ mkdir docs/docs/latest/
+
+ ln -s "../${ICEBERG_VERSION}/docs" docs/docs/latest/docs
+ cp "docs/docs/${ICEBERG_VERSION}/mkdocs.yml" docs/docs/latest/
+
+ cd docs/docs/
+ update_version "latest"
+ cd -
+}
+
+
+# https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/#search-exclusion
+search_exclude_versioned_docs () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ cd "${ICEBERG_VERSION}/docs/"
+
+ python3 -c "import os
Review Comment:
We both use python and python3, we probably want to consolidate this
##########
site/dev/common.sh:
##########
@@ -0,0 +1,125 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+mv_nightly_up () {
+ mv docs/docs/nightly/ docs/
+}
+
+mv_nightly_down () {
+ mv docs/nightly/ docs/docs/
+}
+
+uninstall_deps () {
+ if test -d site_env; then
+ deactivate
+ rm -r site_env
+ fi
+}
+
+install_deps () {
+ uninstall_deps
+ python -m venv site_env
+ source site_env/bin/activate
+ pip install -r requirements.txt
+}
+
+assert_not_empty () {
+ if [ -z "$1" ]
+ then
+ echo "No argument supplied"
+ exit 1
+ fi
+}
+
+get_latest_version () {
+ basename $(ls -d docs/docs/*/ | sort -V | tail -1)
+}
+
+update_version () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ sed -i '' -E "s/(^site\_name:[[:space:]]+docs\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+ sed -i '' -E "s/(^[[:space:]]*-[[:space:]]+Javadoc:.*\/javadoc\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+
+ sed -i '' -E "s/ \- latest: '\!include docs\/docs\/latest\/mkdocs\.yml'/a \- ${ICEBERG_VERSION}: '\!include docs\/docs\/${ICEBERG_VERSION}\/mkdocs\.yml/" ../../mkdocs.yml
+}
+
+create_latest () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ rm -rf docs/docs/latest/
+ mkdir docs/docs/latest/
+
+ ln -s "../${ICEBERG_VERSION}/docs" docs/docs/latest/docs
+ cp "docs/docs/${ICEBERG_VERSION}/mkdocs.yml" docs/docs/latest/
+
+ cd docs/docs/
+ update_version "latest"
+ cd -
+}
+
+
+# https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/#search-exclusion
+search_exclude_versioned_docs () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ cd "${ICEBERG_VERSION}/docs/"
+
+ python3 -c "import os
+for f in filter(lambda x: x.endswith('.md'), os.listdir()): lines = open(f).readlines(); open(f, 'w').writelines(lines[:2] + ['search:\n', ' exclude: true\n'] + lines[2:]);"
+
+ cd -
+}
+
+pull_versioned_docs () {
+ mv_nightly_up
+
+ git worktree add docs/docs docs
+ git worktree add docs/javadoc javadoc
Review Comment:
My `make serve` gets stuck on this one:
```
+ git worktree add docs/docs docs
Preparing worktree (checking out 'docs')
HEAD is now at 87d2dfbd6 Shift site build to use monorepo and gh-pages
+ git worktree add docs/javadoc javadoc
fatal: invalid reference: javadoc
```
##########
site/dev/common.sh:
##########
@@ -0,0 +1,107 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+mv_nightly_up () {
+ mv docs/docs/nightly/ docs/
+}
+
+mv_nightly_down () {
+ mv docs/nightly/ docs/docs/
+}
+
+assert_not_empty () {
+ if [ -z "$1" ]
+ then
+ echo "No argument supplied"
+ exit 1
+ fi
+}
+
+get_latest_version () {
+ basename $(ls -d docs/docs/*/ | sort -V | tail -1)
+}
+
+update_version () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ sed -i '' -E "s/(^site\_name:[[:space:]]+docs\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+ sed -i '' -E "s/(^[[:space:]]*-[[:space:]]+Javadoc:.*\/javadoc\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+}
+
+create_latest () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ rm -rf docs/docs/latest/
+ mkdir docs/docs/latest/
+
+ ln -s "../${ICEBERG_VERSION}/docs" docs/docs/latest/docs
+ cp "docs/docs/${ICEBERG_VERSION}/mkdocs.yml" docs/docs/latest/
+
+ cd docs/docs/
+ update_version "latest"
+ cd -
+}
+
+
+# https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/#search-exclusion
+search_exclude_versioned_docs () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ cd "${ICEBERG_VERSION}/docs/"
+
+ python3 -c "import os
+for f in filter(lambda x: x.endswith('.md'), os.listdir()): lines = open(f).readlines(); open(f, 'w').writelines(lines[:2] + ['search:\n', ' exclude: true\n'] + lines[2:]);"
Review Comment:
Nice! 😁
##########
site/Makefile:
##########
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+.PHONY: help
+help: # Show help for each of the Makefile recipes.
+ @grep -E '^[a-zA-Z0-9 -]+:.*#' Makefile | sort | while read -r l; do printf "\033[1;32m$$(echo $$l | cut -f 1 -d':')\033[00m:$$(echo $$l | cut -f 2- -d'#')\n"; done
+
+.PHONY: serve
+serve: # Clean, build, and run the docs site locally.
+ dev/serve.sh
+
+.PHONY: build
+build: # Clean and build the docs site locally.
Review Comment:
In Makefile you can also invoke other commands, for example here: https://github.com/apache/iceberg-python/blob/main/Makefile#L24
This way we can pull logic from the scripts themselves, into the Makefile (and therefore we don't need this kind of comments).
##########
.github/workflows/site-ci.yml:
##########
@@ -0,0 +1,21 @@
+name: ci
Review Comment:
```suggestion
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
name: ci
```
##########
site/README.md:
##########
@@ -83,59 +99,27 @@ All previously versioned docs will be committed in `docs-<version>` branches and
└── ...
```
-### Install
-
-1. (Optional) Set up venv
+To run this, run the `serve` recipe, which runs the `build` recipe and calls `mkdocs serve`. This will run locally at <http://localhost:8000>.
```
-python -m venv mkdocs_env
-source mkdocs_env/bin/activate
+make serve
```
-1. Install required Python libraries
+To clear all build files, run `clean`.
```
-pip install -r requirements.txt
-```
-
-#### Adding additional versioned documentation
-
-To build locally with additional docs versions, add them to your working tree.
-For now, I'm just adding a single version, and the javadocs directory.
-
-```
-git worktree add site/docs/docs/1.4.0 docs-1.4.0
-git worktree add site/docs/javadoc javadoc
-```
-
-## Build
-
-Run the build command in the root directory, and optionally add `--clean` to force MkDocs to clear previously generated pages.
-
-```
-mkdocs build [--clean]
-```
-
-## Run
-
-Start MkDocs server locally to verify the site looks good.
-
-```
-mkdocs serve
+make clean
```
## Release process
-Deploying a version of the docs is a two step process:
- 1. ~~Cut a new release from the current branch revision. This creates a new branch `docs-<version>`.~~
-
+Deploying the docs is a two step process:
+ 1. Release a new version by copying the current nightly directory to a new version directory in the `docs` branch.
```
- .github/bin/deploy_docs.sh -v 1.4.0
+ make release ICEBERG_VERSION=${ICEBERG_VERSION}
+ ```
+ 1. Push the generated site to `asf-site`.
+ ```
+ make deploy
Review Comment:
Will this be done by a person? We could also do this in the CI, as we do with PyIceberg: https://github.com/apache/iceberg-python/blob/main/.github/workflows/python-ci-docs.yml
##########
site/.gitignore:
##########
@@ -1,13 +1,3 @@
-## Temp remove for first phase
Review Comment:
Do we need this one at all? We can just rely on the main `.gitignore`
##########
site/dev/common.sh:
##########
@@ -0,0 +1,125 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+mv_nightly_up () {
+ mv docs/docs/nightly/ docs/
+}
+
+mv_nightly_down () {
+ mv docs/nightly/ docs/docs/
+}
+
+uninstall_deps () {
+ if test -f .env; then
+ deactivate
+ rm -r .env
+ fi
+}
+
+install_deps () {
+ uninstall_deps
+ python -m venv .env
+ source env/bin/activate
Review Comment:
```suggestion
source .env/bin/activate
```
##########
site/dev/common.sh:
##########
@@ -0,0 +1,107 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+mv_nightly_up () {
+ mv docs/docs/nightly/ docs/
+}
+
+mv_nightly_down () {
+ mv docs/nightly/ docs/docs/
+}
+
+assert_not_empty () {
+ if [ -z "$1" ]
+ then
+ echo "No argument supplied"
+ exit 1
+ fi
+}
+
+get_latest_version () {
+ basename $(ls -d docs/docs/*/ | sort -V | tail -1)
+}
+
+update_version () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ sed -i '' -E "s/(^site\_name:[[:space:]]+docs\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+ sed -i '' -E "s/(^[[:space:]]*-[[:space:]]+Javadoc:.*\/javadoc\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+}
+
+create_latest () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ rm -rf docs/docs/latest/
+ mkdir docs/docs/latest/
+
+ ln -s "../${ICEBERG_VERSION}/docs" docs/docs/latest/docs
+ cp "docs/docs/${ICEBERG_VERSION}/mkdocs.yml" docs/docs/latest/
+
+ cd docs/docs/
+ update_version "latest"
+ cd -
+}
+
+
+# https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/#search-exclusion
+search_exclude_versioned_docs () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ cd "${ICEBERG_VERSION}/docs/"
+
+ python3 -c "import os
+for f in filter(lambda x: x.endswith('.md'), os.listdir()): lines = open(f).readlines(); open(f, 'w').writelines(lines[:2] + ['search:\n', ' exclude: true\n'] + lines[2:]);"
Review Comment:
Nice! 😁
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#issuecomment-1883219914
This looks great, and I don't see any open comments. If there is anything we can follow up on in a separate PR. I'll go ahead and rebase and merge this 🚀 Thanks for the reviews @rdblue, @RussellSpitzer, and a special thanks to @bitsondatadev for taking care of the docs!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1371640541
##########
.github/workflows/spark-ci.yml:
##########
@@ -141,4 +142,4 @@ jobs:
with:
name: test logs
path: |
- **/build/testlogs
\ No newline at end of file
+ **/build/testlogs
Review Comment:
So I just tried removing the file changes from my local and the GH PR, then added this line again as I did in the other files and this artifact keeps popping up in the diff. Any thoughts on how to get rid of it? It's not a newline or even a space or anything. I tried those as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1371717288
##########
.github/workflows/spark-ci.yml:
##########
@@ -141,4 +142,4 @@ jobs:
with:
name: test logs
path: |
- **/build/testlogs
\ No newline at end of file
+ **/build/testlogs
Review Comment:
Looks like the trailing newline was missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1388593968
##########
site/mkdocs.yml:
##########
@@ -23,33 +23,37 @@ theme:
logo: assets/images/iceberg-logo-icon.png
favicon: assets/images/favicon-96x96.png
features:
+ - content.tabs.link
+ - content.code.copy
+ - content.code.annotate
- navigation.tabs
- navigation.path
- navigation.top
- navigation.tracking
- - toc.follow
+ - offline
- search.suggest
- search.highlight
- - content.tabs.link
- - content.code.copy
- - content.code.annotate
+ - toc.follow
plugins:
- search
- macros:
include_yaml:
- variables.yml
- monorepo
+ - offline:
+ enabled: !ENV [OFFLINE, false]
nav:
- Quickstart:
- Spark: spark-quickstart.md
- Hive: hive-quickstart.md
- Docs:
- nightly: '!include docs/docs/nightly/mkdocs.yml'
-# - latest: '!include docs/docs/latest/mkdocs.yml'
-# - 1.3.1: '!include docs/docs/1.3.1/mkdocs.yml'
-# - 1.3.0: '!include docs/docs/1.3.0/mkdocs.yml'
+ - latest: '!include docs/docs/latest/mkdocs.yml'
Review Comment:
I was looking at this:
<img width="1524" alt="image" src="https://github.com/apache/iceberg/assets/1134248/c15926b7-9d7c-4017-9ece-819bea66c930">
It looks a bit odd since 1.4.2 is the same as latest. Do we really need this, it is quite obvious that 1.4.2 is the latest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1391563201
##########
site/README.md:
##########
@@ -35,107 +35,101 @@ In MkDocs, the [`docs_dir`](https://www.mkdocs.org/user-guide/configuration/#doc
### Iceberg docs layout
-In the Iceberg docs, since the top-level site and versioned docs are contained in the same directory, they all live under the `/site` directory of the main Iceberg repository. The `/site/docs` directory is named this way to follow the [MkDocs convention](https://www.mkdocs.org/user-guide/configuration/#docs_dir), while the `/site/docs/docs` directory is an analog to the "Docs" navigation tab. Under this directory, you'll find the `/site/docs/docs/nightly` directory, which contains the state of the documentation in the local revisions.
+The static Iceberg website and versioned documentation live together under the `/site` directory of the main Iceberg repository. The `/site/docs` directory is named that way to follow the [MkDocs convention](https://www.mkdocs.org/user-guide/configuration/#docs_dir), while the nested `/site/docs/docs` directory is an analog to the "Docs" navigation tab. The `/site/docs/nightly` directory contains the current state of the versioned documentation with local revisions.
-The non-versioned site pages are all the `/site/docs/.*md` files and the docs are the `/site/docs/docs/<version>/docs/*.md` files. Notice the location of the `mkdocs.yml`. Looking at this though, you may ask where the older versions and javadocs are.
+
+The static Iceberg site pages are Markdown files that live at `/site/docs/.*md`. The versioned documentation are Markdown files that are added at the nested docs directy `/site/docs/docs/<version>/docs/*.md` files. Notice the location of the `mkdocs.yml`. Looking at this though, you may ask where the older versions and javadocs are.
Review Comment:
Not sure I understand the last two sentences of this paragraph, are they needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #8919:
URL: https://github.com/apache/iceberg/pull/8919
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1391967054
##########
site/mkdocs.yml:
##########
@@ -23,33 +23,37 @@ theme:
logo: assets/images/iceberg-logo-icon.png
favicon: assets/images/favicon-96x96.png
features:
+ - content.tabs.link
+ - content.code.copy
+ - content.code.annotate
- navigation.tabs
- navigation.path
- navigation.top
- navigation.tracking
- - toc.follow
+ - offline
- search.suggest
- search.highlight
- - content.tabs.link
- - content.code.copy
- - content.code.annotate
+ - toc.follow
plugins:
- search
- macros:
include_yaml:
- variables.yml
- monorepo
+ - offline:
+ enabled: !ENV [OFFLINE, false]
nav:
- Quickstart:
- Spark: spark-quickstart.md
- Hive: hive-quickstart.md
- Docs:
- nightly: '!include docs/docs/nightly/mkdocs.yml'
-# - latest: '!include docs/docs/latest/mkdocs.yml'
-# - 1.3.1: '!include docs/docs/1.3.1/mkdocs.yml'
-# - 1.3.0: '!include docs/docs/1.3.0/mkdocs.yml'
+ - latest: '!include docs/docs/latest/mkdocs.yml'
Review Comment:
Yeah, so nightly is unstable. I thought it would be best to have it up top for the easiest access. Perhaps this ordering?:
```
1.4.2 (latest)
nightly
1.4.1
1.4.0
...
```
Although that looks a bit off to me now.
Edit: Perhaps
```
nightly (unstable)
1.4.2 (latest)
1.4.1
1.4.0
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1391967054
##########
site/mkdocs.yml:
##########
@@ -23,33 +23,37 @@ theme:
logo: assets/images/iceberg-logo-icon.png
favicon: assets/images/favicon-96x96.png
features:
+ - content.tabs.link
+ - content.code.copy
+ - content.code.annotate
- navigation.tabs
- navigation.path
- navigation.top
- navigation.tracking
- - toc.follow
+ - offline
- search.suggest
- search.highlight
- - content.tabs.link
- - content.code.copy
- - content.code.annotate
+ - toc.follow
plugins:
- search
- macros:
include_yaml:
- variables.yml
- monorepo
+ - offline:
+ enabled: !ENV [OFFLINE, false]
nav:
- Quickstart:
- Spark: spark-quickstart.md
- Hive: hive-quickstart.md
- Docs:
- nightly: '!include docs/docs/nightly/mkdocs.yml'
-# - latest: '!include docs/docs/latest/mkdocs.yml'
-# - 1.3.1: '!include docs/docs/1.3.1/mkdocs.yml'
-# - 1.3.0: '!include docs/docs/1.3.0/mkdocs.yml'
+ - latest: '!include docs/docs/latest/mkdocs.yml'
Review Comment:
Yeah, so nightly is unstable. I thought it would be best to have it up top for the easiest access. Perhaps this ordering?:
```
1.4.2 (latest)
nightly
1.4.1
1.4.0
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1378313906
##########
site/README.md:
##########
@@ -125,17 +124,16 @@ mkdocs serve
## Release process
Deploying a version of the docs is a two step process:
- 1. ~~Cut a new release from the current branch revision. This creates a new branch `docs-<version>`.~~
+ 1. ~~Release a new version by copying the current nightly directory to a new version directory in the `docs` branch.~~
Review Comment:
https://github.com/apache/iceberg/tree/main/site/docs/docs/nightly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1378313906
##########
site/README.md:
##########
@@ -125,17 +124,16 @@ mkdocs serve
## Release process
Deploying a version of the docs is a two step process:
- 1. ~~Cut a new release from the current branch revision. This creates a new branch `docs-<version>`.~~
+ 1. ~~Release a new version by copying the current nightly directory to a new version directory in the `docs` branch.~~
Review Comment:
https://github.com/apache/iceberg/tree/main/site/docs/docs/nightly
Perhaps I should add a link there? I definitely think all of this explanation docs needs to be reviewed. It was an
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1391589914
##########
site/README.md:
##########
@@ -35,107 +35,101 @@ In MkDocs, the [`docs_dir`](https://www.mkdocs.org/user-guide/configuration/#doc
### Iceberg docs layout
-In the Iceberg docs, since the top-level site and versioned docs are contained in the same directory, they all live under the `/site` directory of the main Iceberg repository. The `/site/docs` directory is named this way to follow the [MkDocs convention](https://www.mkdocs.org/user-guide/configuration/#docs_dir), while the `/site/docs/docs` directory is an analog to the "Docs" navigation tab. Under this directory, you'll find the `/site/docs/docs/nightly` directory, which contains the state of the documentation in the local revisions.
+The static Iceberg website and versioned documentation live together under the `/site` directory of the main Iceberg repository. The `/site/docs` directory is named that way to follow the [MkDocs convention](https://www.mkdocs.org/user-guide/configuration/#docs_dir), while the nested `/site/docs/docs` directory is an analog to the "Docs" navigation tab. The `/site/docs/nightly` directory contains the current state of the versioned documentation with local revisions.
-The non-versioned site pages are all the `/site/docs/.*md` files and the docs are the `/site/docs/docs/<version>/docs/*.md` files. Notice the location of the `mkdocs.yml`. Looking at this though, you may ask where the older versions and javadocs are.
+
+The static Iceberg site pages are Markdown files that live at `/site/docs/.*md`. The versioned documentation are Markdown files that are added at the nested docs directy `/site/docs/docs/<version>/docs/*.md` files. Notice the location of the `mkdocs.yml`. Looking at this though, you may ask where the older versions and javadocs are.
```
./site/
├── docs
│ ├── assets
-│ ├── docs
-│ │ └── nightly
-│ │ ├── docs
-│ │ │ ├── assets
-│ │ │ ├── api.md
-│ │ │ ├── ...
-│ │ │ └── table-migration.md
-│ │ └── mkdocs.yml (versioned)
+│ ├── nightly
+│ │ ├── docs
+│ │ │ ├── assets
+│ │ │ ├── api.md
+│ │ │ ├── ...
+│ │ │ └── table-migration.md
+│ │ └── mkdocs.yml (versioned)
│ ├── about.md
│ ├── ...
│ └── view-spec.md
-├── README.md
+├── ...
+├── Makefile
├── mkdocs.yml (non-versioned)
-├── requirements.txt
-└── variables.yml
+└── requirements.txt
```
### Building the versioned docs
-> [!IMPORTANT]
-> This build process is currently missing older versions and the javadoc branches.
-> Until these branches are merged, these steps will not work.
+The Iceberg versioned docs are committed in the orphan `docs` branch and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The `docs` branch contains the versioned documenation source files at the root. These versions exist at the `/site/docs/docs/<version>` directory once added to the worktree. The `nightly` and `latest` versions, are soft links to the `/site/docs/nightly` directory and greatest release version in the `docs` branch. There is also a `javadoc` branch that contains prior staticly generated versions of the javadocs mounted at `/site/docs/javadoc/<version>`.
+
+The docs are built, run, and released using [make](https://www.gnu.org/software/make/manual/make.html). The [Makefile](Makefile) and the [common shell script](dev/common.sh) support the following command:
+
+``` site > make help```
+> [build](dev/build.sh): Clean and build the site locally.
+> [clean](dev/clean.sh): Clean the local site.
+> [deploy](dev/deploy.sh): Clean, build, and deploy the Iceberg docs site.
+> help: Show help for each of the Makefile recipes.
+> [release](dev/release.sh): Release the current nightly docs as ICEBERG_VERSION (make release ICEBERG_VERSION=<MAJOR.MINOR.PATCH>).
+> [serve](dev/serve.sh): Clean, build, and run the site locally.
-All previously versioned docs will be committed in `docs-<version>` branches and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree will pull these versions in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs in a single tag.
+To scaffold the versioned docs and build the project, run the `build` recipe.
+
+```
+make build
+```
+
+This step will generate the following layout:
```
./site/
└── docs
├── docs
- │ ├── nightly
- │ ├── latest
+ │ ├── nightly (symlink to /site/docs/nightly/)
+ │ ├── latest (symlink to /site/docs/1.4.0/)
│ ├── 1.4.0
│ ├── 1.3.1
│ └── ...
- └── javadoc
- ├── latest
- ├── 1.4.0
- ├── 1.3.1
- └── ...
-```
-
-### Install
-
-1. (Optional) Set up venv
-```
-python -m venv mkdocs_env
-source mkdocs_env/bin/activate
+ ├── javadoc
+ │ ├── latest
+ │ ├── 1.4.0
+ │ ├── 1.3.1
+ │ └── ...
+ └── nightly
```
-1. Install required Python libraries
+To run this, run the `serve` recipe, which runs the `build` recipe and calls `mkdocs serve`. This will run locally at <http://localhost:8000>.
```
-pip install -r requirements.txt
+make serve
```
-#### Adding additional versioned documentation
-
-To build locally with additional docs versions, add them to your working tree.
-For now, I'm just adding a single version, and the javadocs directory.
-
+To clear all build files, run `clean`.
```
-git worktree add site/docs/docs/1.4.0 docs-1.4.0
-git worktree add site/docs/javadoc javadoc
+make clean
```
-## Build
+#### Offline mode
-Run the build command in the root directory, and optionally add `--clean` to force MkDocs to clear previously generated pages.
+One of the great advantages to the MkDocs material plugin is the [offline feature](https://squidfunk.github.io/mkdocs-material/plugins/offline). You can view the Iceberg docs without the need of a server. To enable OFFLINE builds, add theOFFLINE environment variable to either `build` or `serve` recipes.
Review Comment:
missing space "theOFFLINE"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1391628443
##########
site/mkdocs.yml:
##########
@@ -23,33 +23,37 @@ theme:
logo: assets/images/iceberg-logo-icon.png
favicon: assets/images/favicon-96x96.png
features:
+ - content.tabs.link
+ - content.code.copy
+ - content.code.annotate
- navigation.tabs
- navigation.path
- navigation.top
- navigation.tracking
- - toc.follow
+ - offline
- search.suggest
- search.highlight
- - content.tabs.link
- - content.code.copy
- - content.code.annotate
+ - toc.follow
plugins:
- search
- macros:
include_yaml:
- variables.yml
- monorepo
+ - offline:
+ enabled: !ENV [OFFLINE, false]
nav:
- Quickstart:
- Spark: spark-quickstart.md
- Hive: hive-quickstart.md
- Docs:
- nightly: '!include docs/docs/nightly/mkdocs.yml'
-# - latest: '!include docs/docs/latest/mkdocs.yml'
-# - 1.3.1: '!include docs/docs/1.3.1/mkdocs.yml'
-# - 1.3.0: '!include docs/docs/1.3.0/mkdocs.yml'
+ - latest: '!include docs/docs/latest/mkdocs.yml'
Review Comment:
For me I was wondering why "nightly" was first isn't that our "unstable" basically?
I agree ideally we have something like 1.4.2 (latest)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1391561630
##########
site/README.md:
##########
@@ -35,107 +35,101 @@ In MkDocs, the [`docs_dir`](https://www.mkdocs.org/user-guide/configuration/#doc
### Iceberg docs layout
-In the Iceberg docs, since the top-level site and versioned docs are contained in the same directory, they all live under the `/site` directory of the main Iceberg repository. The `/site/docs` directory is named this way to follow the [MkDocs convention](https://www.mkdocs.org/user-guide/configuration/#docs_dir), while the `/site/docs/docs` directory is an analog to the "Docs" navigation tab. Under this directory, you'll find the `/site/docs/docs/nightly` directory, which contains the state of the documentation in the local revisions.
+The static Iceberg website and versioned documentation live together under the `/site` directory of the main Iceberg repository. The `/site/docs` directory is named that way to follow the [MkDocs convention](https://www.mkdocs.org/user-guide/configuration/#docs_dir), while the nested `/site/docs/docs` directory is an analog to the "Docs" navigation tab. The `/site/docs/nightly` directory contains the current state of the versioned documentation with local revisions.
Review Comment:
minor nit: "directory is named that way to follow the" => "is named following the"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1375311435
##########
site/README.md:
##########
@@ -65,7 +65,7 @@ The non-versioned site pages are all the `/site/docs/.*md` files and the docs ar
> This build process is currently missing older versions and the javadoc branches.
> Until these branches are merged, these steps will not work.
-All previously versioned docs will be committed in `docs-<version>` branches and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree will pull these versions in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs in a single tag.
+The Iceberg versioned docs are committed in the `docs` branch and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree pulls these version directories in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs.
Review Comment:
Are all docs in the same branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1378313906
##########
site/README.md:
##########
@@ -125,17 +124,16 @@ mkdocs serve
## Release process
Deploying a version of the docs is a two step process:
- 1. ~~Cut a new release from the current branch revision. This creates a new branch `docs-<version>`.~~
+ 1. ~~Release a new version by copying the current nightly directory to a new version directory in the `docs` branch.~~
Review Comment:
https://github.com/apache/iceberg/tree/main/site/docs/docs/nightly
Perhaps I should add a link there? I definitely think all of this explanation docs needs to be reviewed. It was a quick first pass :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1376385112
##########
site/README.md:
##########
@@ -65,7 +65,7 @@ The non-versioned site pages are all the `/site/docs/.*md` files and the docs ar
> This build process is currently missing older versions and the javadoc branches.
> Until these branches are merged, these steps will not work.
-All previously versioned docs will be committed in `docs-<version>` branches and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree will pull these versions in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs in a single tag.
+The Iceberg versioned docs are committed in the `docs` branch and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree pulls these version directories in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs.
Review Comment:
Yes, please refer to https://github.com/apache/iceberg/tree/docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1386108273
##########
site/docs/docs/nightly/docs/configuration.md:
##########
Review Comment:
All of the changes under `site/docs/docs/**` are all in a seperate commit. This was just to keep the docs up-to-date so I could release Docs for 1.4.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1388588215
##########
site/README.md:
##########
@@ -35,107 +35,101 @@ In MkDocs, the [`docs_dir`](https://www.mkdocs.org/user-guide/configuration/#doc
### Iceberg docs layout
-In the Iceberg docs, since the top-level site and versioned docs are contained in the same directory, they all live under the `/site` directory of the main Iceberg repository. The `/site/docs` directory is named this way to follow the [MkDocs convention](https://www.mkdocs.org/user-guide/configuration/#docs_dir), while the `/site/docs/docs` directory is an analog to the "Docs" navigation tab. Under this directory, you'll find the `/site/docs/docs/nightly` directory, which contains the state of the documentation in the local revisions.
+The static Iceberg website and versioned documentation live together under the `/site` directory of the main Iceberg repository. The `/site/docs` directory is named that way to follow the [MkDocs convention](https://www.mkdocs.org/user-guide/configuration/#docs_dir), while the nested `/site/docs/docs` directory is an analog to the "Docs" navigation tab. The `/site/docs/nightly` directory contains the current state of the versioned documentation with local revisions.
-The non-versioned site pages are all the `/site/docs/.*md` files and the docs are the `/site/docs/docs/<version>/docs/*.md` files. Notice the location of the `mkdocs.yml`. Looking at this though, you may ask where the older versions and javadocs are.
+
+The static Iceberg site pages are Markdown files that live at `/site/docs/.*md`. The versioned documentation are Markdown files that are added at the nested docs directy `/site/docs/docs/<version>/docs/*.md` files. Notice the location of the `mkdocs.yml`. Looking at this though, you may ask where the older versions and javadocs are.
```
./site/
├── docs
│ ├── assets
-│ ├── docs
-│ │ └── nightly
-│ │ ├── docs
-│ │ │ ├── assets
-│ │ │ ├── api.md
-│ │ │ ├── ...
-│ │ │ └── table-migration.md
-│ │ └── mkdocs.yml (versioned)
+│ ├── nightly
+│ │ ├── docs
+│ │ │ ├── assets
+│ │ │ ├── api.md
+│ │ │ ├── ...
+│ │ │ └── table-migration.md
+│ │ └── mkdocs.yml (versioned)
│ ├── about.md
│ ├── ...
│ └── view-spec.md
-├── README.md
+├── ...
+├── Makefile
├── mkdocs.yml (non-versioned)
-├── requirements.txt
-└── variables.yml
+└── requirements.txt
```
### Building the versioned docs
-> [!IMPORTANT]
-> This build process is currently missing older versions and the javadoc branches.
-> Until these branches are merged, these steps will not work.
+The Iceberg versioned docs are committed in the orphan `docs` branch and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The `docs` branch contains the versioned documenation source files at the root. These versions exist at the `/site/docs/docs/<version>` directory once added to the worktree. The `nightly` and `latest` versions, are soft links to the `/site/docs/nightly` directory and greatest release version in the `docs` branch. There is also a `javadoc` branch that contains prior staticly generated versions of the javadocs mounted at `/site/docs/javadoc/<version>`.
+
+The docs are built, run, and released using [make](https://www.gnu.org/software/make/manual/make.html). The [Makefile](Makefile) and the [common shell script](dev/common.sh) support the following command:
+
+``` site > make help```
+> [build](dev/build.sh): Clean and build the site locally.
+> [clean](dev/clean.sh): Clean the local site.
+> [deploy](dev/deploy.sh): Clean, build, and deploy the Iceberg docs site.
+> help: Show help for each of the Makefile recipes.
+> [release](dev/release.sh): Release the current nightly docs as ICEBERG_VERSION (make release ICEBERG_VERSION=<MAJOR.MINOR.PATCH>).
+> [serve](dev/serve.sh): Clean, build, and run the site locally.
-All previously versioned docs will be committed in `docs-<version>` branches and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree will pull these versions in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs in a single tag.
+To scaffold the versioned docs and build the project, run the `build` recipe.
+
+```
+make build
+```
+
+This step will generate the following layout:
```
./site/
└── docs
├── docs
- │ ├── nightly
- │ ├── latest
+ │ ├── nightly (symlink to /site/docs/nightly/)
+ │ ├── latest (symlink to /site/docs/1.4.0/)
Review Comment:
Should we add the 1.4.2 etc here for clarity?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#issuecomment-1804689906
Niiice! Got the `make serve` running in one go now:
<img width="1380" alt="image" src="https://github.com/apache/iceberg/assets/1134248/4d8dd416-48d2-4281-b9e6-f060d6eb8dbd">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1378313906
##########
site/README.md:
##########
@@ -125,17 +124,16 @@ mkdocs serve
## Release process
Deploying a version of the docs is a two step process:
- 1. ~~Cut a new release from the current branch revision. This creates a new branch `docs-<version>`.~~
+ 1. ~~Release a new version by copying the current nightly directory to a new version directory in the `docs` branch.~~
Review Comment:
https://github.com/apache/iceberg/tree/main/site/docs/docs/nightly
Perhaps I should add a link there? I definitely think all of this explanation docs needs to be reviewed. It was a quick first pass, but I do believe that's out of the scope of this PR. This is just a move to get the build working. We'll fix the words once the build has been set up, and we are contributing to one docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1378304225
##########
site/README.md:
##########
@@ -125,17 +124,16 @@ mkdocs serve
## Release process
Deploying a version of the docs is a two step process:
- 1. ~~Cut a new release from the current branch revision. This creates a new branch `docs-<version>`.~~
+ 1. ~~Release a new version by copying the current nightly directory to a new version directory in the `docs` branch.~~
```
.github/bin/deploy_docs.sh -v 1.4.0
Review Comment:
Yeah,
I had a discussion with @rdblue that we should add deploy scripts in this PR and not have the script that I added where if run in the wrong directory did bad things 😬.
https://github.com/apache/iceberg/pull/8659#discussion_r1367561171
##########
site/README.md:
##########
@@ -125,17 +124,16 @@ mkdocs serve
## Release process
Deploying a version of the docs is a two step process:
- 1. ~~Cut a new release from the current branch revision. This creates a new branch `docs-<version>`.~~
+ 1. ~~Release a new version by copying the current nightly directory to a new version directory in the `docs` branch.~~
```
.github/bin/deploy_docs.sh -v 1.4.0
Review Comment:
Yeah,
I had a discussion with @rdblue that we should add deploy scripts in this PR and not have the script that I added where if run in the wrong directory did bad things 😬.
https://github.com/apache/iceberg/pull/8659#discussion_r1367561171
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#issuecomment-1879255453
@Fokko I updated after your comments!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1443356487
##########
site/docs/community.md:
##########
@@ -40,13 +40,13 @@ Issues are tracked in GitHub:
## Slack
-We use the [Apache Iceberg workspace](https://apache-iceberg.slack.com/) on Slack. To be invited, follow [this invite link](https://join.slack.com/t/apache-iceberg/shared_invite/zt-1znkcg5zm-7_FE~pcox347XwZE3GNfPg).
+We use the [Apache Iceberg workspace](https://apache-iceberg.slack.com/) on Slack. To be invited, follow [this invite link](https://join.slack.com/t/apache-iceberg/shared_invite/zt-287g3akar-K9Oe_En5j1UL7Y_Ikpai3A).
Review Comment:
Oh yeah, I still need to figure out how to embed jitsi stuff there, another day.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1386219088
##########
site/dev/common.sh:
##########
@@ -0,0 +1,125 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+mv_nightly_up () {
+ mv docs/docs/nightly/ docs/
+}
+
+mv_nightly_down () {
+ mv docs/nightly/ docs/docs/
+}
+
+uninstall_deps () {
+ if test -f .env; then
+ deactivate
+ rm -r .env
+ fi
+}
+
+install_deps () {
+ uninstall_deps
+ python -m venv .env
+ source env/bin/activate
+ pip install -r requirements.txt
+}
+
+assert_not_empty () {
+ if [ -z "$1" ]
+ then
+ echo "No argument supplied"
+ exit 1
+ fi
+}
+
+get_latest_version () {
+ basename $(ls -d docs/docs/*/ | sort -V | tail -1)
+}
+
+update_version () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ sed -i '' -E "s/(^site\_name:[[:space:]]+docs\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+ sed -i '' -E "s/(^[[:space:]]*-[[:space:]]+Javadoc:.*\/javadoc\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+
+ sed -i '' -E "s/(^ \- latest: '\!include docs\/docs\/latest\/mkdocs\.yml'/a \- ${ICEBERG_VERSION}: '\!include docs\/docs\/${ICEBERG_VERSION}\/mkdocs\.yml/" ../../mkdocs.yml
+}
+
+create_latest () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ rm -rf docs/docs/latest/
+ mkdir docs/docs/latest/
+
+ ln -s "../${ICEBERG_VERSION}/docs" docs/docs/latest/docs
+ cp "docs/docs/${ICEBERG_VERSION}/mkdocs.yml" docs/docs/latest/
+
+ cd docs/docs/
+ update_version "latest"
+ cd -
+}
+
+
+# https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/#search-exclusion
+search_exclude_versioned_docs () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ cd "${ICEBERG_VERSION}/docs/"
+
+ python3 -c "import os
Review Comment:
Yes, I prefer `python3`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1386213158
##########
site/README.md:
##########
@@ -83,59 +99,27 @@ All previously versioned docs will be committed in `docs-<version>` branches and
└── ...
```
-### Install
-
-1. (Optional) Set up venv
+To run this, run the `serve` recipe, which runs the `build` recipe and calls `mkdocs serve`. This will run locally at <http://localhost:8000>.
```
-python -m venv mkdocs_env
-source mkdocs_env/bin/activate
+make serve
```
-1. Install required Python libraries
+To clear all build files, run `clean`.
```
-pip install -r requirements.txt
-```
-
-#### Adding additional versioned documentation
-
-To build locally with additional docs versions, add them to your working tree.
-For now, I'm just adding a single version, and the javadocs directory.
-
-```
-git worktree add site/docs/docs/1.4.0 docs-1.4.0
-git worktree add site/docs/javadoc javadoc
-```
-
-## Build
-
-Run the build command in the root directory, and optionally add `--clean` to force MkDocs to clear previously generated pages.
-
-```
-mkdocs build [--clean]
-```
-
-## Run
-
-Start MkDocs server locally to verify the site looks good.
-
-```
-mkdocs serve
+make clean
```
## Release process
-Deploying a version of the docs is a two step process:
- 1. ~~Cut a new release from the current branch revision. This creates a new branch `docs-<version>`.~~
-
+Deploying the docs is a two step process:
+ 1. Release a new version by copying the current nightly directory to a new version directory in the `docs` branch.
```
- .github/bin/deploy_docs.sh -v 1.4.0
+ make release ICEBERG_VERSION=${ICEBERG_VERSION}
+ ```
+ 1. Push the generated site to `asf-site`.
+ ```
+ make deploy
Review Comment:
No, I guess I should remove this as that basically documents this step. I'll only keep build and serve documented in the readme.
Although one last thought is that I just want all this documented so that others can read through to understand my intent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1386210561
##########
site/Makefile:
##########
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+.PHONY: help
+help: # Show help for each of the Makefile recipes.
+ @grep -E '^[a-zA-Z0-9 -]+:.*#' Makefile | sort | while read -r l; do printf "\033[1;32m$$(echo $$l | cut -f 1 -d':')\033[00m:$$(echo $$l | cut -f 2- -d'#')\n"; done
+
+.PHONY: serve
+serve: # Clean, build, and run the docs site locally.
+ dev/serve.sh
+
+.PHONY: build
+build: # Clean and build the docs site locally.
Review Comment:
lol, actually, the comments were a hack for `make help` or default `make` output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1443340509
##########
site/docs/community.md:
##########
@@ -40,13 +40,13 @@ Issues are tracked in GitHub:
## Slack
-We use the [Apache Iceberg workspace](https://apache-iceberg.slack.com/) on Slack. To be invited, follow [this invite link](https://join.slack.com/t/apache-iceberg/shared_invite/zt-1znkcg5zm-7_FE~pcox347XwZE3GNfPg).
+We use the [Apache Iceberg workspace](https://apache-iceberg.slack.com/) on Slack. To be invited, follow [this invite link](https://join.slack.com/t/apache-iceberg/shared_invite/zt-287g3akar-K9Oe_En5j1UL7Y_Ikpai3A).
Review Comment:
I checked and this is the latest one: https://github.com/apache/iceberg-docs/commit/87149e3da04eafad1c8766132f0bf1e822e4d075
##########
.github/workflows/site-ci.yml:
##########
@@ -1,3 +1,4 @@
+#
Review Comment:
Looks like git sees this as a move 🤔 Not a real problem.
##########
site/nav.yml:
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+nav:
+ - Home: index.md
+ - Quickstart:
+ - Spark: spark-quickstart.md
+ - Hive: hive-quickstart.md
+ - Docs:
+ #- nightly: '!include docs/docs/nightly/mkdocs.yml'
+ - latest: '!include docs/docs/latest/mkdocs.yml'
+ - 1.4.2: '!include docs/docs/1.4.2/mkdocs.yml'
Review Comment:
I think 1.4.3 is missing here.
##########
site/docs/spec.md:
##########
@@ -1128,6 +1125,41 @@ Example
] } ]
```
+### Content File (Data and Delete) Serialization
Review Comment:
Not for this PR, but it would be good at some point to just include the spec from the repository itself.
##########
site/docs/roadmap.md:
##########
@@ -20,28 +20,37 @@ title: "Roadmap"
# Roadmap Overview
-This roadmap outlines projects that the Iceberg community is working on, their priority, and a rough size estimate.
-This is based on the latest [community priority discussion](https://lists.apache.org/thread.html/r84e80216c259c81f824c6971504c321cd8c785774c489d52d4fc123f%40%3Cdev.iceberg.apache.org%3E).
+This roadmap outlines projects that the Iceberg community is working on.
Each high-level item links to a Github project board that tracks the current status.
Related design docs will be linked on the planning boards.
-# Priority 1
-
-* API: [Iceberg 1.0.0](https://github.com/apache/iceberg/projects/3) [medium]
-* Python: [Pythonic refactor](https://github.com/apache/iceberg/projects/7) [medium]
-* Spec: [Z-ordering / Space-filling curves](https://github.com/apache/iceberg/projects/16) [medium]
-* Spec: [Snapshot tagging and branching](https://github.com/apache/iceberg/projects/4) [small]
-* Views: [Spec](https://github.com/apache/iceberg/projects/6) [medium]
-* Puffin: [Implement statistics information in table snapshot](https://github.com/apache/iceberg/pull/4741) [medium]
-* Flink: [FLIP-27 based Iceberg source](https://github.com/apache/iceberg/projects/23) [large]
-
-# Priority 2
-
-* ORC: [Support delete files stored as ORC](https://github.com/apache/iceberg/projects/13) [small]
-* Spark: [DSv2 streaming improvements](https://github.com/apache/iceberg/projects/2) [small]
-* Flink: [Inline file compaction](https://github.com/apache/iceberg/projects/14) [small]
-* Flink: [Support UPSERT](https://github.com/apache/iceberg/projects/15) [small]
-* Spec: [Secondary indexes](https://github.com/apache/iceberg/projects/17) [large]
-* Spec v3: [Encryption](https://github.com/apache/iceberg/projects/5) [large]
-* Spec v3: [Relative paths](https://github.com/apache/iceberg/projects/18) [large]
-* Spec v3: [Default field values](https://github.com/apache/iceberg/projects/19) [medium]
+# General
+
+* [Multi-table transaction support](https://github.com/apache/iceberg/projects/30)
+* [Views Support](https://github.com/apache/iceberg/projects/29)
+* [Change Data Capture (CDC) Support](https://github.com/apache/iceberg/projects/26)
+* [Snapshot tagging and branching](https://github.com/apache/iceberg/projects/4)
+* [Inline file compaction](https://github.com/apache/iceberg/projects/14)
+* [Delete File compaction](https://github.com/apache/iceberg/projects/10)
+* [Z-ordering / Space-filling curves](https://github.com/apache/iceberg/projects/16)
+* [Support UPSERT](https://github.com/apache/iceberg/projects/15)
+
+# Clients
+_Rust and Go projects are pointing to their respective repositories which include
Review Comment:
```suggestion
_Python, Rust and Go projects are pointing to their respective repositories which include
```
##########
site/dev/common.sh:
##########
@@ -0,0 +1,228 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+REMOTE="iceberg_docs"
+
+# Ensures the presence of a specified remote repository for documentation.
+# If the remote doesn't exist, it adds it using the provided URL.
+# Then, it fetches updates from the remote repository.
+create_or_update_docs_remote () {
+ echo " --> create or update docs remote"
+
+ # Check if the remote exists before attempting to add it
+ git config "remote.${REMOTE}.url" >/dev/null ||
+ git remote add "${REMOTE}" https://github.com/apache/iceberg.git
+
+ # Fetch updates from the remote repository
+ git fetch "${REMOTE}"
+}
+
+
+# Pulls updates from a specified branch of a remote repository.
+# Arguments:
+# $1: Branch name to pull updates from
+pull_remote () {
+ echo " --> pull remote"
+
+ local BRANCH="$1"
+
+ # Ensure the branch argument is not empty
+ assert_not_empty "${BRANCH}"
+
+ # Perform a pull from the specified branch of the remote repository
+ git pull "${REMOTE}" "${BRANCH}"
+}
+
+# Pushes changes from a local branch to a specified branch of a remote repository.
+# Arguments:
+# $1: Branch name to push changes to
+push_remote () {
+ echo " --> push remote"
+
+ local BRANCH="$1"
+
+ # Ensure the branch argument is not empty
+ assert_not_empty "${BRANCH}"
+
+ # Push changes to the specified branch of the remote repository
+ git push "${REMOTE}" "${BRANCH}"
+}
+
+# Installs or upgrades dependencies specified in the 'requirements.txt' file using pip.
+install_deps () {
+ echo " --> install deps"
+
+ # Use pip to install or upgrade dependencies from the 'requirements.txt' file quietly
+ pip -q install -r requirements.txt --upgrade
+}
+
+# Checks if a provided argument is not empty. If empty, displays an error message and exits with a status code 1.
+# Arguments:
+# $1: Argument to check for emptiness
+assert_not_empty () {
+
+ if [ -z "$1" ]; then
+ echo "No argument supplied"
+
+ # Exit with an error code if no argument is provided
+ exit 1
+ fi
+}
+
+# Finds and retrieves the latest version of the documentation based on the directory structure.
+# Assumes the documentation versions are numeric folders within 'docs/docs/'.
+get_latest_version () {
+ # Find the latest numeric folder within 'docs/docs/' structure
+ local latest=$(ls -d docs/docs/[0-9]* | sort -V | tail -1)
+
+ # Extract the version number from the latest directory path
+ local latest_version=$(basename "${latest}")
+
+ # Output the latest version number
+ echo "${latest_version}"
+}
+
+# Creates a symbolic link for a 'nightly' version of the documentation.
+create_nightly () {
+ echo " --> create nightly"
+
+ # Remove any existing 'nightly' symbolic link to prevent conflicts
+ rm -f docs/docs/nightly/
+
+ # Create a symbolic link pointing to the 'nightly' documentation
+ ln -s ../nightly ../docs
+}
+
+# Creates a 'latest' version of the documentation based on a specified ICEBERG_VERSION.
+# Arguments:
+# $1: ICEBERG_VERSION - The version number of the documentation to be treated as the latest.
+create_latest () {
+ echo " --> create latest"
+
+ local ICEBERG_VERSION="$1"
+
+ # Ensure ICEBERG_VERSION is not empty
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ # Output the provided ICEBERG_VERSION for verification
+ echo "${ICEBERG_VERSION}"
+
+ # Remove any existing 'latest' directory and recreate it
+ rm -rf docs/docs/latest/
+ mkdir docs/docs/latest/
+
+ # Create symbolic links and copy configuration files for the 'latest' documentation
+ ln -s "../${ICEBERG_VERSION}/docs" docs/docs/latest/docs
+ cp "docs/docs/${ICEBERG_VERSION}/mkdocs.yml" docs/docs/latest/
+
+ cd docs/docs/
+
+ # Update version information within the 'latest' documentation
+ update_version "latest"
+ cd -
+}
+
+# Updates version information within the mkdocs.yml file for a specified ICEBERG_VERSION.
+# Arguments:
+# $1: ICEBERG_VERSION - The version number used for updating the mkdocs.yml file.
+update_version () {
+ echo " --> update version"
+
+ local ICEBERG_VERSION="$1"
+
+ # Ensure ICEBERG_VERSION is not empty
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ # Update version information within the mkdocs.yml file using sed commands
+ if [ "$(uname)" == "Darwin" ]
+ then
+ sed -i '' -E "s/(^site\_name:[[:space:]]+docs\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+ sed -i '' -E "s/(^[[:space:]]*-[[:space:]]+Javadoc:.*\/javadoc\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+ elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]
+ then
+ sed -i'' -E "s/(^site_name:[[:space:]]+docs\/)[^[:space:]]+/\1${ICEBERG_VERSION}/" "${ICEBERG_VERSION}/mkdocs.yml"
+ sed -i'' -E "s/(^[[:space:]]*-[[:space:]]+Javadoc:.*\/javadoc\/).*$/\1${ICEBERG_VERSION}/" "${ICEBERG_VERSION}/mkdocs.yml"
+ fi
+
+}
+
+# Excludes versioned documentation from search indexing by modifying .md files.
+# Arguments:
+# $1: ICEBERG_VERSION - The version number of the documentation to exclude from search indexing.
+search_exclude_versioned_docs () {
+ echo " --> search exclude version docs"
+ local ICEBERG_VERSION="$1"
+
+ # Ensure ICEBERG_VERSION is not empty
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ cd "${ICEBERG_VERSION}/docs/"
+
+ # Modify .md files to exclude versioned documentation from search indexing
+ python3 -c "import os
+for f in filter(lambda x: x.endswith('.md'), os.listdir()): lines = open(f).readlines(); open(f, 'w').writelines(lines[:2] + ['search:\n', ' exclude: true\n'] + lines[2:]);"
+
+ cd -
+}
+
+# Sets up local worktrees for the documentation and performs operations related to different versions.
+pull_versioned_docs () {
+ echo " --> pull versioned docs"
+
+ # Ensure the remote repository for documentation exists and is up-to-date
+ create_or_update_docs_remote
+
+ rm -r docs/docs
+
+ # Add local worktrees for documentation and javadoc from the remote repository
+ git worktree add docs/docs "${REMOTE}/docs"
+ git worktree add docs/javadoc "${REMOTE}/javadoc"
Review Comment:
I had still some old cruft on my worktree, so I needed to do this:
```suggestion
git worktree add -f docs/docs "${REMOTE}/docs"
git worktree add -f docs/javadoc "${REMOTE}/javadoc"
```
##########
site/docs/releases.md:
##########
@@ -64,7 +67,140 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`:
</dependencies>
```
-## 1.3.1 release
+### 1.4.3 Release
+
+Apache Iceberg 1.4.3 was released on December 27, 2023. The main issue it solves is missing files from a transaction retry with conflicting manifests. It is recommended to upgrade if you use transactions.
+
+- Core: Scan only live entries in partitions table (#8969) by @Fokko in [#9197](https://github.com/apache/iceberg/pull/9197)
+- Core: Fix missing files from transaction retries with conflicting manifest merges by [@nastra](https://github.com/nastra) in [#9337]O(https://github.com/apache/iceberg/pull/9337)
Review Comment:
This was caught by @manuzhang in https://github.com/apache/iceberg-docs/pull/299 🙌
```suggestion
- Core: Fix missing files from transaction retries with conflicting manifest merges by [@nastra](https://github.com/nastra) in [#9337](https://github.com/apache/iceberg/pull/9337)
```
##########
.github/workflows/site-ci.yml:
##########
@@ -14,13 +15,24 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-extra:
- icebergVersion: 1.4.0
- social:
- - icon: fontawesome/brands/github-alt
- link: https://github.com/apache/iceberg
- - icon: fontawesome/brands/youtube
- link: https://www.youtube.com/@ApacheIceberg
- - icon: fontawesome/brands/slack
- link: https://join.slack.com/t/apache-iceberg/shared_invite/zt-1znkcg5zm-7_FE~pcox347XwZE3GNfPg
+#
+name: site-ci
+on:
+ push:
+ branches:
+ - main
+ paths:
+ - site/**
+ workflow_dispatch:
+jobs:
+ deploy:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v3
+ - uses: actions/setup-python@v4
+ with:
+ python-version: 3.x
+ - name: Deploy Iceberg documentation
+ run: |
+ make deploy
Review Comment:
Nit:
```suggestion
run: make deploy
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1443359408
##########
site/nav.yml:
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+nav:
+ - Home: index.md
+ - Quickstart:
+ - Spark: spark-quickstart.md
+ - Hive: hive-quickstart.md
+ - Docs:
+ #- nightly: '!include docs/docs/nightly/mkdocs.yml'
+ - latest: '!include docs/docs/latest/mkdocs.yml'
+ - 1.4.2: '!include docs/docs/1.4.2/mkdocs.yml'
Review Comment:
I will update 1.4.3 on another PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1375311282
##########
.github/labeler.yml:
##########
@@ -32,6 +32,7 @@ BUILD:
- "**/*gradle*"
DOCS:
- docs/**/*
+ - site/**/*
Review Comment:
Can we fix this in a separate PR? It isn't necessary for this one and it would be great to get it in faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1389541886
##########
site/README.md:
##########
@@ -35,107 +35,101 @@ In MkDocs, the [`docs_dir`](https://www.mkdocs.org/user-guide/configuration/#doc
### Iceberg docs layout
-In the Iceberg docs, since the top-level site and versioned docs are contained in the same directory, they all live under the `/site` directory of the main Iceberg repository. The `/site/docs` directory is named this way to follow the [MkDocs convention](https://www.mkdocs.org/user-guide/configuration/#docs_dir), while the `/site/docs/docs` directory is an analog to the "Docs" navigation tab. Under this directory, you'll find the `/site/docs/docs/nightly` directory, which contains the state of the documentation in the local revisions.
+The static Iceberg website and versioned documentation live together under the `/site` directory of the main Iceberg repository. The `/site/docs` directory is named that way to follow the [MkDocs convention](https://www.mkdocs.org/user-guide/configuration/#docs_dir), while the nested `/site/docs/docs` directory is an analog to the "Docs" navigation tab. The `/site/docs/nightly` directory contains the current state of the versioned documentation with local revisions.
-The non-versioned site pages are all the `/site/docs/.*md` files and the docs are the `/site/docs/docs/<version>/docs/*.md` files. Notice the location of the `mkdocs.yml`. Looking at this though, you may ask where the older versions and javadocs are.
+
+The static Iceberg site pages are Markdown files that live at `/site/docs/.*md`. The versioned documentation are Markdown files that are added at the nested docs directy `/site/docs/docs/<version>/docs/*.md` files. Notice the location of the `mkdocs.yml`. Looking at this though, you may ask where the older versions and javadocs are.
```
./site/
├── docs
│ ├── assets
-│ ├── docs
-│ │ └── nightly
-│ │ ├── docs
-│ │ │ ├── assets
-│ │ │ ├── api.md
-│ │ │ ├── ...
-│ │ │ └── table-migration.md
-│ │ └── mkdocs.yml (versioned)
+│ ├── nightly
+│ │ ├── docs
+│ │ │ ├── assets
+│ │ │ ├── api.md
+│ │ │ ├── ...
+│ │ │ └── table-migration.md
+│ │ └── mkdocs.yml (versioned)
│ ├── about.md
│ ├── ...
│ └── view-spec.md
-├── README.md
+├── ...
+├── Makefile
├── mkdocs.yml (non-versioned)
-├── requirements.txt
-└── variables.yml
+└── requirements.txt
```
### Building the versioned docs
-> [!IMPORTANT]
-> This build process is currently missing older versions and the javadoc branches.
-> Until these branches are merged, these steps will not work.
+The Iceberg versioned docs are committed in the orphan `docs` branch and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The `docs` branch contains the versioned documenation source files at the root. These versions exist at the `/site/docs/docs/<version>` directory once added to the worktree. The `nightly` and `latest` versions, are soft links to the `/site/docs/nightly` directory and greatest release version in the `docs` branch. There is also a `javadoc` branch that contains prior staticly generated versions of the javadocs mounted at `/site/docs/javadoc/<version>`.
+
+The docs are built, run, and released using [make](https://www.gnu.org/software/make/manual/make.html). The [Makefile](Makefile) and the [common shell script](dev/common.sh) support the following command:
+
+``` site > make help```
+> [build](dev/build.sh): Clean and build the site locally.
+> [clean](dev/clean.sh): Clean the local site.
+> [deploy](dev/deploy.sh): Clean, build, and deploy the Iceberg docs site.
+> help: Show help for each of the Makefile recipes.
+> [release](dev/release.sh): Release the current nightly docs as ICEBERG_VERSION (make release ICEBERG_VERSION=<MAJOR.MINOR.PATCH>).
+> [serve](dev/serve.sh): Clean, build, and run the site locally.
-All previously versioned docs will be committed in `docs-<version>` branches and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree will pull these versions in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs in a single tag.
+To scaffold the versioned docs and build the project, run the `build` recipe.
+
+```
+make build
+```
+
+This step will generate the following layout:
```
./site/
└── docs
├── docs
- │ ├── nightly
- │ ├── latest
+ │ ├── nightly (symlink to /site/docs/nightly/)
+ │ ├── latest (symlink to /site/docs/1.4.0/)
Review Comment:
Yeah, that or we could just use some variable placeholder representation in the README to represent the latest version: `$LATEST` or something. My only concern is people might read it too literally.
Maybe we can do the good ol' `<latest_version>` or like pseudo mustache syntax `{latest_version}`.
Any preference?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1391967054
##########
site/mkdocs.yml:
##########
@@ -23,33 +23,37 @@ theme:
logo: assets/images/iceberg-logo-icon.png
favicon: assets/images/favicon-96x96.png
features:
+ - content.tabs.link
+ - content.code.copy
+ - content.code.annotate
- navigation.tabs
- navigation.path
- navigation.top
- navigation.tracking
- - toc.follow
+ - offline
- search.suggest
- search.highlight
- - content.tabs.link
- - content.code.copy
- - content.code.annotate
+ - toc.follow
plugins:
- search
- macros:
include_yaml:
- variables.yml
- monorepo
+ - offline:
+ enabled: !ENV [OFFLINE, false]
nav:
- Quickstart:
- Spark: spark-quickstart.md
- Hive: hive-quickstart.md
- Docs:
- nightly: '!include docs/docs/nightly/mkdocs.yml'
-# - latest: '!include docs/docs/latest/mkdocs.yml'
-# - 1.3.1: '!include docs/docs/1.3.1/mkdocs.yml'
-# - 1.3.0: '!include docs/docs/1.3.0/mkdocs.yml'
+ - latest: '!include docs/docs/latest/mkdocs.yml'
Review Comment:
Yeah, so nightly is unstable. I thought it would be best to have it up top for the easiest access. Perhaps this ordering?:
```
1.4.2 (latest)
nightly
1.4.1
1.4.0
...
```
Although that looks a bit off to me now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1378313362
##########
site/README.md:
##########
@@ -65,7 +65,7 @@ The non-versioned site pages are all the `/site/docs/.*md` files and the docs ar
> This build process is currently missing older versions and the javadoc branches.
> Until these branches are merged, these steps will not work.
-All previously versioned docs will be committed in `docs-<version>` branches and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree will pull these versions in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs in a single tag.
+The Iceberg versioned docs are committed in the `docs` branch and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree pulls these version directories in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs.
Review Comment:
Not that this isn't correct English but as a native speaker, it reads almost like you're saying the convention is what pulls the directories in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #8919:
URL: https://github.com/apache/iceberg/pull/8919#discussion_r1386524118
##########
site/dev/common.sh:
##########
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+mv_nightly_up () {
+ mv docs/docs/nightly/ docs/
+}
+
+mv_nightly_down () {
+ mv docs/nightly/ docs/docs/
+}
+
+install_deps () {
+ pip install -r requirements.txt --upgrade
+}
+
+assert_not_empty () {
+ if [ -z "$1" ]
+ then
+ echo "No argument supplied"
+ exit 1
+ fi
+}
+
+get_latest_version () {
+ basename $(ls -d docs/docs/*/ | sort -V | tail -1)
+}
+
+update_version () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ sed -i '' -E "s/(^site\_name:[[:space:]]+docs\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+ sed -i '' -E "s/(^[[:space:]]*-[[:space:]]+Javadoc:.*\/javadoc\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml
+
+ #sed -i '' -E "s/ \- latest: '\!include docs\/docs\/latest\/mkdocs\.yml'/a \- ${ICEBERG_VERSION}: '\!include docs\/docs\/${ICEBERG_VERSION}\/mkdocs\.yml/" ../../mkdocs.yml
+}
+
+create_latest () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ rm -rf docs/docs/latest/
+ mkdir docs/docs/latest/
+
+ ln -s "../${ICEBERG_VERSION}/docs" docs/docs/latest/docs
+ cp "docs/docs/${ICEBERG_VERSION}/mkdocs.yml" docs/docs/latest/
+
+ cd docs/docs/
+ update_version "latest"
+ cd -
+}
+
+
+# https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/#search-exclusion
+search_exclude_versioned_docs () {
+ local ICEBERG_VERSION="$1"
+ assert_not_empty "${ICEBERG_VERSION}"
+
+ cd "${ICEBERG_VERSION}/docs/"
+
+ python3 -c "import os
+for f in filter(lambda x: x.endswith('.md'), os.listdir()): lines = open(f).readlines(); open(f, 'w').writelines(lines[:2] + ['search:\n', ' exclude: true\n'] + lines[2:]);"
+
+ cd -
+}
+
+pull_versioned_docs () {
+ mv_nightly_up
+
+ git worktree add docs/docs docs
+ git worktree add docs/javadoc javadoc
Review Comment:
```suggestion
git remote add iceberg-docs https://github.com/apache/iceberg.git
git remote fetch iceberg-docs
git worktree add docs/docs iceberg-docs/docs
git worktree add docs/javadoc iceberg-docs/javadoc
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org