You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@polaris.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2024/09/15 19:53:08 UTC
[PR] Spark Jupyter getting started docker compose [polaris]
kevinjqliu opened a new pull request, #295:
URL: https://github.com/apache/polaris/pull/295
# Description
This PR moves the `docker-compose-jupyter.yml` file (and the `notebooks/` directory), formerly in the top-level directory, into the `getting-started/spark/` folder.
The purpose is to unify the "getting started" guides into the same directory.
Fixes #110
## Type of change
Please delete options that are not relevant.
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] Documentation update
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [x] This change requires a documentation update
# How Has This Been Tested?
```
docker-compose -f getting-started/spark/docker-compose-jupyter.yml up
```
# Checklist:
Please delete options that are not relevant.
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] If adding new functionality, I have discussed my implementation with the community using the linked GitHub issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #295:
URL: https://github.com/apache/polaris/pull/295#discussion_r1777604556
##########
getting-started/spark/create-polaris-catalog.sh:
##########
Review Comment:
might be, if you want a spark-shell. I think the jupyter notebook does a good job of explaining a lot of the concepts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #295:
URL: https://github.com/apache/polaris/pull/295#discussion_r1777610035
##########
getting-started/spark/README.md:
##########
@@ -0,0 +1,51 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Spark and Apache Polaris
+
+This getting started guide provides a `docker-compose` file to set up [Spark](https://spark.apache.org/) with Apache Polaris. Apache Polaris is configured as an Iceberg REST Catalog in Spark.
+A Jupyter notebook is used to run PySpark.
+
+## Run the `docker-compose` file
+To start the `docker-compose` file, run this command from the repo's root directory:
+```
+docker-compose -f getting-started/spark/docker-compose.yml up
+```
+
+This will spin up 3 containers services
+* The `polaris` service for running Apache Polaris
+* The `jupyter` service for running Jupyter notebook with PySpark
+* The `create-polaris-catalog` service to run setup script and create local catalog in Polaris
+
+## Access the Jupyter notebook interface
+In the Jupyter notebook container log, look for the URL to access Jupyter notebook. The url should be in the format, `http://127.0.0.1:8888/lab?token=<token>`.
Review Comment:
changed to `to access the Jupyter notebook`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #295:
URL: https://github.com/apache/polaris/pull/295#issuecomment-2353450842
Sounds good. We will need these doc to be in the Polaris doc site, like this https://polaris.apache.org/docs/overview/. I couldn't find Trino's doc there, this may involve doc publish and link. cc @jbonofre
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #295:
URL: https://github.com/apache/polaris/pull/295#discussion_r1801804488
##########
getting-started/spark/create-polaris-catalog.sh:
##########
@@ -0,0 +1,56 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:default-realm}"
+POLARIS_CATALOG_NAME="${POLARIS_CATALOG_NAME:-polaris_demo}"
+
+# create a catalog backed by the local filesystem
Review Comment:
We could initialize catalog in notebook as well. I feel it's more flexible that way. for example, you don't have to worry about the an env variable for catalog name. But I'm OK with either one. Not a blocker for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #295:
URL: https://github.com/apache/polaris/pull/295#discussion_r1801792907
##########
getting-started/spark/create-polaris-catalog.sh:
##########
@@ -0,0 +1,56 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:default-realm}"
+POLARIS_CATALOG_NAME="${POLARIS_CATALOG_NAME:-polaris_demo}"
+
+# create a catalog backed by the local filesystem
Review Comment:
we could, but i think its a good idea to separate infra code (this script) from application code (the notebook)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #295:
URL: https://github.com/apache/polaris/pull/295#issuecomment-2353431656
@flyrain yep i'll have a README in here, similar to the trino one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #295:
URL: https://github.com/apache/polaris/pull/295#issuecomment-2351771121
I want to make sure this is something we want to do before proceeding to add more to the PR
cc @collado-mike / @flyrain
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #295:
URL: https://github.com/apache/polaris/pull/295#issuecomment-2353608507
@collado-mike makes sense, will do.
I have a question [on slack](https://apache-iceberg.slack.com/archives/C07HADW86HZ/p1726508953511979) about unable to assume the role `arn:aws:iam::631484165566:role/datalake-storage-integration-role` in the notebook, do you mind taking a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #295:
URL: https://github.com/apache/polaris/pull/295#discussion_r1777606833
##########
getting-started/spark/README.md:
##########
@@ -0,0 +1,51 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Spark and Apache Polaris
+
+This getting started guide provides a `docker-compose` file to set up [Spark](https://spark.apache.org/) with Apache Polaris. Apache Polaris is configured as an Iceberg REST Catalog in Spark.
+A Jupyter notebook is used to run PySpark.
+
+## Run the `docker-compose` file
+To start the `docker-compose` file, run this command from the repo's root directory:
+```
+docker-compose -f getting-started/spark/docker-compose.yml up
+```
+
+This will spin up 3 containers services
+* The `polaris` service for running Apache Polaris
+* The `jupyter` service for running Jupyter notebook with PySpark
+* The `create-polaris-catalog` service to run setup script and create local catalog in Polaris
+
+## Access the Jupyter notebook interface
+In the Jupyter notebook container log, look for the URL to access Jupyter notebook. The url should be in the format, `http://127.0.0.1:8888/lab?token=<token>`.
Review Comment:
which part is the typo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain merged PR #295:
URL: https://github.com/apache/polaris/pull/295
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #295:
URL: https://github.com/apache/polaris/pull/295#issuecomment-2414840472
Thanks for the review @flyrain, addressed your comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #295:
URL: https://github.com/apache/polaris/pull/295#issuecomment-2415001468
Thanks for the heads up, I'll rebase once that PR's merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #295:
URL: https://github.com/apache/polaris/pull/295#issuecomment-2414975095
We cannot merge any PR until #374 is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #295:
URL: https://github.com/apache/polaris/pull/295#discussion_r1801731763
##########
getting-started/spark/README.md:
##########
@@ -0,0 +1,51 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Spark and Apache Polaris
+
+This getting started guide provides a `docker-compose` file to set up [Apache Spark](https://spark.apache.org/) with Apache Polaris. Apache Polaris is configured as an Iceberg REST Catalog in Spark.
+A Jupyter notebook is used to run PySpark.
+
+## Run the `docker-compose` file
+To start the `docker-compose` file, run this command from the repo's root directory:
+```
+docker-compose -f getting-started/spark/docker-compose.yml up
+```
+
+This will spin up 3 container services
+* The `polaris` service for running Apache Polaris
+* The `jupyter` service for running Jupyter notebook with PySpark
+* The `create-polaris-catalog` service to run setup script and create local catalog in Polaris
Review Comment:
`local catalog` -> `a catalog backed by the local file system`?
##########
getting-started/spark/README.md:
##########
@@ -0,0 +1,51 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Spark and Apache Polaris
+
+This getting started guide provides a `docker-compose` file to set up [Apache Spark](https://spark.apache.org/) with Apache Polaris. Apache Polaris is configured as an Iceberg REST Catalog in Spark.
+A Jupyter notebook is used to run PySpark.
+
+## Run the `docker-compose` file
+To start the `docker-compose` file, run this command from the repo's root directory:
+```
+docker-compose -f getting-started/spark/docker-compose.yml up
+```
+
+This will spin up 3 container services
+* The `polaris` service for running Apache Polaris
Review Comment:
Nit: could we be more explicit that it starts with an in-memory metastore?
##########
getting-started/spark/create-polaris-catalog.sh:
##########
@@ -0,0 +1,56 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:default-realm}"
+POLARIS_CATALOG_NAME="${POLARIS_CATALOG_NAME:-polaris_demo}"
+
+# create a catalog backed by the local filesystem
Review Comment:
I'm not entirely sure if we need this file. Could we handle everything directly within the notebook, like the other operations in `SparkPolaris.ipynb`? Would it simplify things if we moved the operations there?
##########
getting-started/spark/README.md:
##########
@@ -0,0 +1,51 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Spark and Apache Polaris
+
+This getting started guide provides a `docker-compose` file to set up [Apache Spark](https://spark.apache.org/) with Apache Polaris. Apache Polaris is configured as an Iceberg REST Catalog in Spark.
Review Comment:
There is other way to try Spark with Polaris without docker, it's a not a blocker, we can add it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "collado-mike (via GitHub)" <gi...@apache.org>.
collado-mike commented on PR #295:
URL: https://github.com/apache/polaris/pull/295#issuecomment-2353589223
This looks good to me. We should change the name of the compose file to just `docker-compose.yml` so we don't have to specify the filename in the command line :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #295:
URL: https://github.com/apache/polaris/pull/295#issuecomment-2353459743
I see, this is the [README for trino](https://github.com/apache/polaris/blob/main/getting-started/trino/README.md). I'll add a similar README for spark.
As a follow-up, we can change the Polaris doc to refer to these guides https://polaris.apache.org/docs/quickstart
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #295:
URL: https://github.com/apache/polaris/pull/295#discussion_r1801372121
##########
.github/workflows/check-md-link.yml:
##########
@@ -41,5 +41,5 @@ jobs:
with:
use-quiet-mode: 'yes'
config-file: '.github/workflows/check-md-link-config.json'
- folder-path: 'regtests, regtests/client/python/docs, regtests/client/python, .github, build-logic, polaris-core, polaris-service, extension, spec, k8, notebooks'
Review Comment:
This PR moved `notebooks/` from top-level directory into the `getting-started/` directory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #295:
URL: https://github.com/apache/polaris/pull/295#issuecomment-2374710687
r? @flyrain @RussellSpitzer @collado-mike
Also opened #319 to update the Polaris doc site once this is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #295:
URL: https://github.com/apache/polaris/pull/295#issuecomment-2415236290
Thanks a lot for working on it, @kevinjqliu! Thanks all for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "sfc-gh-emaynard (via GitHub)" <gi...@apache.org>.
sfc-gh-emaynard commented on code in PR #295:
URL: https://github.com/apache/polaris/pull/295#discussion_r1777607868
##########
getting-started/spark/create-polaris-catalog.sh:
##########
Review Comment:
Sorry, I meant the `polaris` CLI instead of using `curl`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "sfc-gh-emaynard (via GitHub)" <gi...@apache.org>.
sfc-gh-emaynard commented on code in PR #295:
URL: https://github.com/apache/polaris/pull/295#discussion_r1777608413
##########
getting-started/spark/README.md:
##########
@@ -0,0 +1,51 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Spark and Apache Polaris
+
+This getting started guide provides a `docker-compose` file to set up [Spark](https://spark.apache.org/) with Apache Polaris. Apache Polaris is configured as an Iceberg REST Catalog in Spark.
+A Jupyter notebook is used to run PySpark.
+
+## Run the `docker-compose` file
+To start the `docker-compose` file, run this command from the repo's root directory:
+```
+docker-compose -f getting-started/spark/docker-compose.yml up
+```
+
+This will spin up 3 containers services
+* The `polaris` service for running Apache Polaris
+* The `jupyter` service for running Jupyter notebook with PySpark
+* The `create-polaris-catalog` service to run setup script and create local catalog in Polaris
+
+## Access the Jupyter notebook interface
+In the Jupyter notebook container log, look for the URL to access Jupyter notebook. The url should be in the format, `http://127.0.0.1:8888/lab?token=<token>`.
Review Comment:
The existing text just says `to access Jupyter notebook`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #295:
URL: https://github.com/apache/polaris/pull/295#issuecomment-2351943158
Make sense to me. Thanks @kevinjqliu! Do we have any doc for its usage? We may add doc if not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "sfc-gh-emaynard (via GitHub)" <gi...@apache.org>.
sfc-gh-emaynard commented on code in PR #295:
URL: https://github.com/apache/polaris/pull/295#discussion_r1777594352
##########
getting-started/spark/README.md:
##########
@@ -0,0 +1,51 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Spark and Apache Polaris
+
+This getting started guide provides a `docker-compose` file to set up [Spark](https://spark.apache.org/) with Apache Polaris. Apache Polaris is configured as an Iceberg REST Catalog in Spark.
Review Comment:
Let's continue to use `Apache Spark` when mentioning Spark the first time. And let's also leave on the `(Incubating)` for Polaris for now. Looking forward to a day when we can remove it though :)
##########
getting-started/spark/README.md:
##########
@@ -0,0 +1,51 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Spark and Apache Polaris
+
+This getting started guide provides a `docker-compose` file to set up [Spark](https://spark.apache.org/) with Apache Polaris. Apache Polaris is configured as an Iceberg REST Catalog in Spark.
+A Jupyter notebook is used to run PySpark.
+
+## Run the `docker-compose` file
+To start the `docker-compose` file, run this command from the repo's root directory:
+```
+docker-compose -f getting-started/spark/docker-compose.yml up
+```
+
+This will spin up 3 containers services
Review Comment:
typo: `container services`
##########
getting-started/spark/create-polaris-catalog.sh:
##########
Review Comment:
I wonder if it might be easier to use the CLI here
##########
getting-started/spark/README.md:
##########
Review Comment:
I'm a bit conflicted about this doc. It feels like it doesn't really _teach_ the reader anything about Polaris, although it does give you a really fast way to get bootstrapped.
##########
getting-started/spark/README.md:
##########
@@ -0,0 +1,51 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Spark and Apache Polaris
+
+This getting started guide provides a `docker-compose` file to set up [Spark](https://spark.apache.org/) with Apache Polaris. Apache Polaris is configured as an Iceberg REST Catalog in Spark.
+A Jupyter notebook is used to run PySpark.
+
+## Run the `docker-compose` file
+To start the `docker-compose` file, run this command from the repo's root directory:
+```
+docker-compose -f getting-started/spark/docker-compose.yml up
+```
+
+This will spin up 3 containers services
+* The `polaris` service for running Apache Polaris
+* The `jupyter` service for running Jupyter notebook with PySpark
+* The `create-polaris-catalog` service to run setup script and create local catalog in Polaris
+
+## Access the Jupyter notebook interface
+In the Jupyter notebook container log, look for the URL to access Jupyter notebook. The url should be in the format, `http://127.0.0.1:8888/lab?token=<token>`.
Review Comment:
typo: `the Jupyter notebook`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #295:
URL: https://github.com/apache/polaris/pull/295#discussion_r1777609079
##########
getting-started/spark/create-polaris-catalog.sh:
##########
Review Comment:
ah i dont know how to use the polaris CLI, so i just copied directly from https://github.com/apache/polaris/blob/main/regtests/run_spark_sql.sh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #295:
URL: https://github.com/apache/polaris/pull/295#discussion_r1777605346
##########
getting-started/spark/README.md:
##########
Review Comment:
yes I'll admit, this README is a filler for now as a way to get spark & polaris up and running quickly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Spark Jupyter getting started docker compose [polaris]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #295:
URL: https://github.com/apache/polaris/pull/295#issuecomment-2415222050
@flyrain took your advice, moved `getting-started/spark/create-polaris-catalog.sh` logic into the jupyter notebook.
Also rebased off latest main. I think this PR is good to go. Please take a look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org