You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhouyuan (via GitHub)" <gi...@apache.org> on 2026/01/22 15:49:35 UTC

[PR] [VL] Adding cache for m2 repo [incubator-gluten]

zhouyuan opened a new pull request, #11469:
URL: https://github.com/apache/incubator-gluten/pull/11469

   <!--
   Thank you for submitting a pull request! Here are some tips:
   
   1. For first-time contributors, please read our contributing guide:
      https://github.com/apache/incubator-gluten/blob/main/CONTRIBUTING.md
   2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work.
   3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the
      Velox or ClickHouse backend, respectively.
   4. If the PR is not ready for review, please mark it as a draft.
   -->
   
   ## What changes are proposed in this pull request?
   This patch adds the cache for m2 repo after TPC test
   
   
   <!--
   Provide a clear and concise description of the changes introduced in this PR.
   Ensure the PR description aligns with the code changes, especially after updates.
   If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the corresponding issue
   when the PR is merged.
   -->
   
   ## How was this patch tested?
   pass GHA
   
   <!--
   Describe how the changes were tested, if applicable.
   Include new tests to validate the functionality, if necessary.
   For UI-related changes, attach screenshots to demonstrate the updates.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Adding cache for m2 repo [incubator-gluten]

Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #11469:
URL: https://github.com/apache/incubator-gluten/pull/11469#issuecomment-3785566649

   @baibaichen The Java caching can work in this way, however it's too big for now and may trigger evict on the CPP cache
   <img width="1703" height="903" alt="image" src="https://github.com/user-attachments/assets/eb630eb7-ee04-447e-8b65-4d1f4b43a269" />
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Adding cache for m2 repo [incubator-gluten]

Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan closed pull request #11469: [VL] Adding cache for m2 repo
URL: https://github.com/apache/incubator-gluten/pull/11469


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Adding cache for m2 repo [incubator-gluten]

Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on PR #11469:
URL: https://github.com/apache/incubator-gluten/pull/11469#issuecomment-3796455995

   > > How is the cache generated? Is there a shared disk that we mount for each test?
   > 
   > https://github.com/apache/incubator-gluten/actions/caches. You can't mount it. You can copy it. The whole cache size is limited
   
   I see. I agree with proposal of @FelixYBW. We can set up a daily or weekly job to generate the .m2 cache.
   
   **Note**: Since Java Jar files are independent of the OS and Java version, we only need to maintain a single, unified cache for .m2.
   
   To populate this cache comprehensively, we can execute the following commands. Theoretically, this sequence should download all necessary dependencies for our various build profiles into the cache:
   
   ```shell
   build/mvn -P java-17,spark-4.0,scala-2.13,backends-velox,hadoop-3.3,spark-ut -Piceberg,iceberg-test,delta,paimon clean test-compile
   build/mvn -P java-17,spark-3.5,backends-velox,hadoop-3.3,spark-ut -Piceberg,iceberg-test,delta,paimon clean test-compile
   build/mvn -P java-17,spark-3.4,backends-velox,hadoop-3.3,spark-ut -Piceberg,iceberg-test,delta,paimon clean test-compile
   build/mvn -P java-17,spark-3.3,backends-velox,hadoop-3.3,spark-ut -Piceberg,delta,paimon clean test-compile
   build/mvn -P java-17,spark-4.1,scala-2.13,backends-velox,hadoop-3.3,spark-ut clean test-compile
   ```
   
   Regarding `.ccache`, do we currently update it only when upgrading Velox?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Adding cache for m2 repo [incubator-gluten]

Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on PR #11469:
URL: https://github.com/apache/incubator-gluten/pull/11469#issuecomment-3790462351

   How is the cache generated? Is there a shared disk that we mount for each test?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Adding cache for m2 repo [incubator-gluten]

Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #11469:
URL: https://github.com/apache/incubator-gluten/pull/11469#issuecomment-3792112343

   > How is the cache generated? Is there a shared disk that we mount for each test?
   
   https://github.com/apache/incubator-gluten/actions/caches. You can't mount it. You can copy it. The whole cache size is limited


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Adding cache for m2 repo [incubator-gluten]

Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #11469:
URL: https://github.com/apache/incubator-gluten/pull/11469#issuecomment-3787031838

   we may create an image including cache daily, then pull the image and copy .w2 from it. Is it feasible?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Adding cache for m2 repo [incubator-gluten]

Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #11469:
URL: https://github.com/apache/incubator-gluten/pull/11469#issuecomment-3807691851

   fired one 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org