You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "talatuyarer (via GitHub)" <gi...@apache.org> on 2025/10/07 01:15:47 UTC

[PR] Add Fast Mode for Documentation Builds [iceberg]

talatuyarer opened a new pull request, #14267:
URL: https://github.com/apache/iceberg/pull/14267

   Every time we build the docs, the `mkdocs-monorepo-plugin` rebuilds all 14+ versioned documentation of projects even though these old versions never change. This makes iterative development very slow.
   
   I created A development mode that only builds nightly and latest versions to make builds are fast. The production mode still builds all versions which means slow :) 
   
   Based on my machine performance it makes 5x more faster. `--dirty` flag makes even faster iterative rebuilds
   
   #### How to use 
   ***Dev Mod***
   ```shell
   cd site
   make serve-dev  # Only builds nightly + latest
   ```
   ***Full production mode***
   ```shell
   cd site
   make serve      # Builds all 14 versions (use before PRs)
   ```
   
   Closes #14124 
   
   cc @kevinjqliu @RussellSpitzer 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add Fast Mode for Documentation Builds [iceberg]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #14267:
URL: https://github.com/apache/iceberg/pull/14267#discussion_r2499738101


##########
site/mkdocs-dev.yml:
##########
@@ -0,0 +1,107 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a development navigation configuration that only includes
+# nightly and latest documentation for faster build times during
+# iterative development. Use this with ICEBERG_DEV_MODE=true.
+
+INHERIT: ./mkdocs.yml
+
+nav:
+  - Home: index.md
+  - Quickstart:
+    - Spark: spark-quickstart.md
+    - Hive: hive-quickstart.md
+  - Docs:
+    - Java:
+      - Nightly: '!include docs/docs/nightly/mkdocs.yml'
+      - Latest (1.10.0): '!include docs/docs/latest/mkdocs.yml'
+    - Other Implementations:

Review Comment:
   I'm wondering whether we can avoid copying other parts of `mkdocs.yml`. It will be easier to maintain.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add Fast Mode for Documentation Builds [iceberg]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #14267:
URL: https://github.com/apache/iceberg/pull/14267#discussion_r2499894329


##########
site/mkdocs-dev.yml:
##########
@@ -0,0 +1,107 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a development navigation configuration that only includes
+# nightly and latest documentation for faster build times during
+# iterative development. Use this with ICEBERG_DEV_MODE=true.
+
+INHERIT: ./mkdocs.yml
+
+nav:
+  - Home: index.md
+  - Quickstart:
+    - Spark: spark-quickstart.md
+    - Hive: hive-quickstart.md
+  - Docs:
+    - Java:
+      - Nightly: '!include docs/docs/nightly/mkdocs.yml'
+      - Latest (1.10.0): '!include docs/docs/latest/mkdocs.yml'
+    - Other Implementations:

Review Comment:
   agreed, this is more of a short term solution. both `nightly` and `latest` are just sym links to the actual "latest version" (1.10). 
   
   long term i want to explore using [mike](https://github.com/jimporter/mike) for versioning 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add Fast Mode for Documentation Builds [iceberg]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #14267:
URL: https://github.com/apache/iceberg/pull/14267#issuecomment-3493632451

   @talatuyarer pushed up some of my local changes, as we discussed offline. 
   
   Please take a look and see if this is also working for you locally 😄 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add Fast Mode for Documentation Builds [iceberg]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #14267:
URL: https://github.com/apache/iceberg/pull/14267


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add Fast Mode for Documentation Builds [iceberg]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #14267:
URL: https://github.com/apache/iceberg/pull/14267#discussion_r2462962833


##########
site/dev/common.sh:
##########


Review Comment:
   looks like we're still pulling all the versions; i see them in `site/docs/docs`. 
   
   mkdocs seems to be still parsing through all the folders even though the reference is removed in `nav-dev.yml`. 
   
   This is a one of the major issues with slow builds. In https://github.com/apache/iceberg/pull/14154/files#diff-c629cce095c9a6fc10c7c3eedb3f57970f6cd3600f767d5cf333e6493176a29fR23-R29, I had to explicitly remove them 



##########
site/dev/setup_env.sh:
##########
@@ -27,3 +27,14 @@ install_deps
 pull_versioned_docs
 
 git show "${REMOTE}/main:../.asf.yaml" > docs/.asf.yaml
+
+# If running in dev mode, temporarily use the dev navigation file
+if [ "${ICEBERG_DEV_MODE:-false}" = "true" ]; then
+  echo " --> Using dev navigation (nav-dev.yml) for faster builds"
+  # Create a backup of the original nav.yml
+  if [ ! -f nav.yml.backup ]; then
+    cp nav.yml nav.yml.backup
+  fi
+  # Use the dev nav file
+  cp nav-dev.yml nav.yml

Review Comment:
   i see that `clean.sh` is suppose to put the backup back but it has to be called it explicitly after calling `make serve-dev`



##########
site/dev/setup_env.sh:
##########
@@ -27,3 +27,14 @@ install_deps
 pull_versioned_docs
 
 git show "${REMOTE}/main:../.asf.yaml" > docs/.asf.yaml
+
+# If running in dev mode, temporarily use the dev navigation file
+if [ "${ICEBERG_DEV_MODE:-false}" = "true" ]; then
+  echo " --> Using dev navigation (nav-dev.yml) for faster builds"
+  # Create a backup of the original nav.yml
+  if [ ! -f nav.yml.backup ]; then
+    cp nav.yml nav.yml.backup
+  fi
+  # Use the dev nav file
+  cp nav-dev.yml nav.yml

Review Comment:
   nit: i dont love the idea that we're overwriting `nav.yml`. 
   
   Feels like theres a potential footgun here to check in the the `nav.yml` file after its been replaced by `nav-dev.yml`
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add Fast Mode for Documentation Builds [iceberg]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #14267:
URL: https://github.com/apache/iceberg/pull/14267#issuecomment-3497984681

   > INFO    -  Documentation built in 9.59 seconds
   
   supa fast! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add Fast Mode for Documentation Builds [iceberg]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #14267:
URL: https://github.com/apache/iceberg/pull/14267#issuecomment-3497992516

   Thanks @talatuyarer !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org