You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@zeppelin.apache.org by "miinhho (via GitHub)" <gi...@apache.org> on 2026/09/20 08:19:55 UTC
[PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
miinhho opened a new pull request, #5492:
URL: https://github.com/apache/zeppelin/pull/5492
### What is this PR for?
Linux CI jobs repeatedly resolve the same Python 3.9 Conda dependencies during environment setup. This PR makes those jobs install from explicit Linux x86-64 Conda specifications instead, and pins the PyFlink packages installed with pip.
It also separates Maven, npm, Conda, and pip package caches by dependency type. The Livy job now caches the Spark distribution at the path used by its download script. Lock update and validation instructions are documented in `testing/locks/README.md`.
This targets dependency setup time.
CI jobs with those changes applied: https://github.com/miinhho/zeppelin/actions/runs/35498755575
### What type of PR is it?
Improvement
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-6717
### How should this be tested?
- Run the affected Linux CI jobs and confirm they install from the committed Conda specifications without dependency resolution.
- Confirm both PyFlink jobs install their pinned requirements and pass.
- Run CI again to check package-cache reuse and the Livy Spark distribution cache hit.
- Compare environment-setup durations separately from test durations.
### Questions
- License files: No.
- Breaking changes for older versions: No product-runtime change. The explicit Conda specifications are Linux x86-64-specific.
- Documentation: Lock maintenance instructions are included.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "voidmatcha (via GitHub)" <gi...@apache.org>.
voidmatcha commented on code in PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#discussion_r4217348680
##########
testing/locks/python-3.9-linux-64.conda.lock:
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+# created-by: conda 26.7.1
Review Comment:
Sounds good, please go ahead with that approach. Resolving fresh dependencies and validating them before refreshing the locks should address both the staleness and the delayed upstream-breakage detection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#discussion_r4209241807
##########
testing/locks/python-3.9-linux-64.conda.lock:
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+# created-by: conda 26.7.1
Review Comment:
> I confirmed that this PR substantially improves dependency setup time.
I wonder how fast it gains. TBH, I feel this will become stale over time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#discussion_r4209264366
##########
testing/locks/python-3.9-linux-64.conda.lock:
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+# created-by: conda 26.7.1
Review Comment:
and one advantage of not using lock file, is that we can observe the library's upstream breaking changes quickly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "voidmatcha (via GitHub)" <gi...@apache.org>.
voidmatcha merged PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "voidmatcha (via GitHub)" <gi...@apache.org>.
voidmatcha commented on code in PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#discussion_r4209786203
##########
testing/locks/python-3.9-linux-64.conda.lock:
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+# created-by: conda 26.7.1
Review Comment:
@pan3793 Thanks for pointing this out. I compared the same 15 jobs in the [before](https://github.com/apache/zeppelin/actions/runs/36156823942) and [after](https://github.com/apache/zeppelin/actions/runs/36156884258) CI runs. Median environment setup time dropped from 7m00s to 1m02s, roughly an 85% reduction. This includes the added package-cache restore steps and separate PyFlink installation. These are historical CI observations rather than a controlled benchmark, and the reduction applies to environment setup, not total test time.
I agree with your concerns about maintenance and detecting upstream breaking changes. The documentation currently covers manual regeneration, but there is no automated refresh or scheduled check against freshly resolved dependencies. I'll open a follow-up issue to cover periodically resolving dependencies from the YAML files, testing compatibility, and maintaining the lock files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "voidmatcha (via GitHub)" <gi...@apache.org>.
voidmatcha commented on PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#issuecomment-5750188603
I confirmed that this PR substantially improves dependency setup time. As a small follow-up, I put together an [example change](https://github.com/voidmatcha/zeppelin/compare/0b23836cd414bcb974ab9001f06562e32ac2992b...fix/pr5492-spark-cache-version-key) that includes the Spark and Hadoop versions in the cache key and restore prefix, so a future version change cannot keep hitting the previous cache key. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "voidmatcha (via GitHub)" <gi...@apache.org>.
voidmatcha commented on PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#issuecomment-5835290328
Merged into master (33f1fff8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "miinhho (via GitHub)" <gi...@apache.org>.
miinhho commented on PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#issuecomment-5749916860
While working on this PR, I noticed that core.yml repeats several setup steps across jobs. I put together a [follow-up commit](https://github.com/miinhho/zeppelin/commit/a4e09719842ec0fe175fcf93b0c97487a62c890d) that uses [YAML anchors](https://docs.github.com/en/actions/reference/workflows-and-actions/reusing-workflow-configurations#yaml-anchors-and-aliases) to reduce that duplication without changing the expanded job configuration.
I plan to propose it as a separate PR after this one is merged, so this PR can stay focused on dependency setup time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "voidmatcha (via GitHub)" <gi...@apache.org>.
voidmatcha commented on code in PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#discussion_r4236428876
##########
testing/locks/python-3.9-linux-64.conda.lock:
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+# created-by: conda 26.7.1
Review Comment:
Thank you for looking at this so carefully. I overlooked the Apple Silicon workflow when reviewing #5492. The README currently documents lock generation on Ubuntu 24.04 x86-64, but not how to do it from an Apple Silicon Mac.
@miinhho, when updating the maintenance documentation for ZEPPELIN-6739, could you consider including an example of generating and validating the locks in a Docker `linux/amd64` environment? This is not a requirement for the scheduled refresh itself, so if it does not fit that issue, we can track it as a small documentation follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "voidmatcha (via GitHub)" <gi...@apache.org>.
voidmatcha commented on code in PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#discussion_r4209847785
##########
testing/locks/python-3.9-linux-64.conda.lock:
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+# created-by: conda 26.7.1
Review Comment:
@pan3793 Thanks for raising this. Comparing the same 15 CI jobs before and after the merge, median environment setup time dropped from 7m00s to 1m02s, about an 85% reduction. This includes the added cache restore and PyFlink installation steps. These are historical CI observations, not a controlled benchmark.
I agree with the concerns about stale locks and delayed detection of upstream breaking changes. @miinhho, could you look into a follow-up approach, such as scheduled compatibility checks with freshly resolved dependencies and a lock refresh process?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "miinhho (via GitHub)" <gi...@apache.org>.
miinhho commented on code in PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#discussion_r4218638290
##########
testing/locks/python-3.9-linux-64.conda.lock:
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+# created-by: conda 26.7.1
Review Comment:
Thanks! I opened [ZEPPELIN-6739](https://issues.apache.org/jira/browse/ZEPPELIN-6739) to track this follow-up and will start working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "miinhho (via GitHub)" <gi...@apache.org>.
miinhho commented on code in PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#discussion_r4210081634
##########
testing/locks/python-3.9-linux-64.conda.lock:
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+# created-by: conda 26.7.1
Review Comment:
Thanks for sharing the timing comparison. Building on your suggestion, perhaps we could combine periodic lock refreshes with compatibility checks: resolve fresh dependencies and validate them before updating the locks. That could help keep the locks current and catch upstream issues while retaining the setup-time savings.
What do you think? If that sounds good, I’ll look into this approach for a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#discussion_r4236265585
##########
testing/locks/python-3.9-linux-64.conda.lock:
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+# created-by: conda 26.7.1
Review Comment:
another issue found: developers may use macOS with Apple Silicon, while the README instruction says it should use Ubuntu 26.04 with x86_64 processor to generate the conda lock file, I managed to use Docker to simulate x86_64 platform and run it in an Ubuntu container to achieve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "voidmatcha (via GitHub)" <gi...@apache.org>.
voidmatcha commented on code in PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#discussion_r4209786203
##########
testing/locks/python-3.9-linux-64.conda.lock:
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+# created-by: conda 26.7.1
Review Comment:
@pan3793 Thanks for pointing this out. I compared the same 15 jobs in the [before](https://github.com/apache/zeppelin/actions/runs/36156823942) and [after](https://github.com/apache/zeppelin/actions/runs/36156884258) CI runs. Median environment setup time dropped from 7m00s to 1m02s, roughly an 85% reduction. This includes the added package-cache restore steps and separate PyFlink installation. These are historical CI observations rather than a controlled benchmark, and the reduction applies to environment setup, not total test time.
I agree with your concerns about maintenance and detecting upstream breaking changes. The documentation currently covers manual regeneration, but there is no automated refresh or scheduled check against freshly resolved dependencies. I'll open a follow-up issue to cover periodically resolving dependencies from the YAML files, testing compatibility, and maintaining the lock files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#discussion_r4236265585
##########
testing/locks/python-3.9-linux-64.conda.lock:
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+# created-by: conda 26.7.1
Review Comment:
another issue found: developers may use macOS with Apple Silicon, while the README instruction says it should use Ubuntu 26.04 with x86_64 processor to generate the conda lock file, I managed to use Docker to simulate x86_64 platform and run it in an Ubuntu container when working on https://github.com/apache/zeppelin/pull/5465 to achieve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6717] Install locked Python dependencies in Linux CI [zeppelin]
Posted by "miinhho (via GitHub)" <gi...@apache.org>.
miinhho commented on code in PR #5492:
URL: https://github.com/apache/zeppelin/pull/5492#discussion_r4236482155
##########
testing/locks/python-3.9-linux-64.conda.lock:
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+# created-by: conda 26.7.1
Review Comment:
@voidmatcha Agreed, documenting how to generate and validate the locks using Docker linux/amd64 would address the immediate gap for contributors.
Longer term, I think we can simplify the process so contributors do not need to prepare a Docker environment just to generate locks. I looked at pandas’ [pixi.toml](https://github.com/pandas-dev/pandas/blob/main/pixi.toml): it defines the test environments and target platforms, while Pixi generates the committed `pixi.lock`. Pixi resolves those declared platforms without first installing the target environments.
For Zeppelin, we could define our Python test environments in the same way, allowing contributors to update the definitions and run pixi lock locally instead of creating an Ubuntu x86-64 environment and manually exporting the package lists.
This would also help with the periodic refresh work in ZEPPELIN-6739. pandas has a [weekly workflow](https://github.com/pandas-dev/pandas/blob/main/.github/workflows/update-pixi-lock.yml) that runs pixi update and opens a lock-update PR. We could use the same approach for scheduled refreshes, with CI validating the generated changes while regular runs continue installing locked dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org