You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2024/12/22 02:02:26 UTC
[PR] [docs] Replace examples of Hadoop catalog with JDBC catalog [iceberg]
kevinjqliu opened a new pull request, #11845:
URL: https://github.com/apache/iceberg/pull/11845
Closes #11284
This PR replaces examples of Hadoop catalog with the JDBC catalog
[devlist discussion](https://lists.apache.org/thread/2too7vobf4f4h99qonbl2ok06slsww43)
## Testing
### `spark-quickstart.md`
With `spark-sql` CLI config:
```
spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.local.type=jdbc \
--conf spark.sql.catalog.local.uri=jdbc:sqlite:$PWD/iceberg_catalog_db.sqlite \
--conf spark.sql.catalog.local.warehouse=$PWD/warehouse \
--conf spark.sql.defaultCatalog=local
```
With `spark-defaults.conf` file:
```
spark-sql --properties-file ./spark-defaults.conf
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [docs] Replace examples of Hadoop catalog with JDBC & REST catalog [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #11845:
URL: https://github.com/apache/iceberg/pull/11845#discussion_r1935822049
##########
docs/docs/spark-getting-started.md:
##########
@@ -39,22 +39,28 @@ spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ iceb
If you want to include Iceberg in your Spark installation, add the [`iceberg-spark-runtime-3.5_2.12` Jar](https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/{{ icebergVersion }}/iceberg-spark-runtime-3.5_2.12-{{ icebergVersion }}.jar) to Spark's `jars` folder.
-### Adding catalogs
+### Catalogs
-Iceberg comes with [catalogs](spark-configuration.md#catalogs) that enable SQL commands to manage tables and load them by name. Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`.
+Apache Iceberg uses [catalogs](https://iceberg.apache.org/concepts/catalog/) to track and manage table metadata and enable SQL operations. Iceberg provides several [catalog implementations](spark-configuration.md#catalogs), which can be configured using properties under `spark.sql.catalog.(catalog_name)`.
-This command creates a path-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog:
+This command creates a JDBC-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
Review Comment:
maybe something like this,
```suggestion
This command creates a JDBC-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog (`spark_catalog`) using the Hive connector.
```
Is it the "Hive connector" or the "Hive Metastore"?
But I'm also incline not to add this. I feel like this is too detailed for a "getting started" page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [docs] Replace examples of Hadoop catalog with JDBC & REST catalog [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11845:
URL: https://github.com/apache/iceberg/pull/11845#issuecomment-2709153287
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [docs] Replace examples of Hadoop catalog with JDBC & REST catalog [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #11845: [docs] Replace examples of Hadoop catalog with JDBC & REST catalog
URL: https://github.com/apache/iceberg/pull/11845
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [docs] Replace examples of Hadoop catalog with JDBC catalog [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #11845:
URL: https://github.com/apache/iceberg/pull/11845#discussion_r1896903736
##########
site/docs/spark-quickstart.md:
##########
@@ -267,44 +271,109 @@ To read a table, simply use the Iceberg table's name.
df = spark.table("demo.nyc.taxis").show()
```
-### Adding A Catalog
+### Adding catalogs
-Iceberg has several catalog back-ends that can be used to track tables, like JDBC, Hive MetaStore and Glue.
-Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`. In this guide,
-we use JDBC, but you can follow these instructions to configure other catalog types. To learn more, check out
-the [Catalog](docs/latest/spark-configuration.md#catalogs) page in the Spark section.
+Apache Iceberg provides several catalog implementations to manage tables and enable SQL operations.
+Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`.
+You can configure different catalog types, such as JDBC, Hive Metastore, Glue, and REST, to manage Iceberg tables in Spark.
-This configuration creates a path-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
+This guide covers the configuration of two popular catalog types:
+
+* JDBC Catalog
+* REST Catalog
+
+To learn more, check out the [Catalog](docs/latest/spark-configuration.md#catalogs) page in the Spark section.
+
+#### Configuring JDBC Catalog
+
+The JDBC catalog stores Iceberg table metadata in a relational database.
+
+This configuration creates a JDBC-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
+
+The JDBC catalog uses file-based SQLite database as the backend.
=== "CLI"
```sh
- spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}\
+ spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3 \
Review Comment:
taking on this extra dep since i dont see any iceberg specific package i can use. there is a [`hive-jdbc`](https://mvnrepository.com/artifact/org.apache.hive/hive-jdbc) package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [docs] Replace examples of Hadoop catalog with JDBC catalog [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #11845:
URL: https://github.com/apache/iceberg/pull/11845#discussion_r1913035248
##########
docs/docs/spark-getting-started.md:
##########
@@ -41,20 +41,27 @@ spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ iceb
### Adding catalogs
-Iceberg comes with [catalogs](spark-configuration.md#catalogs) that enable SQL commands to manage tables and load them by name. Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`.
+Apache Iceberg provides several [catalog](spark-configuration.md#catalogs) implementations to manage tables and enable SQL operations.
Review Comment:
Before linking to the Spark configuration, how about linking to the Catalogs concept page (https://iceberg.apache.org/concepts/catalog/) to educate the reader on the importance of a catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [docs] Replace examples of Hadoop catalog with JDBC catalog [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #11845:
URL: https://github.com/apache/iceberg/pull/11845#discussion_r1913039173
##########
site/docs/spark-quickstart.md:
##########
@@ -267,44 +271,109 @@ To read a table, simply use the Iceberg table's name.
df = spark.table("demo.nyc.taxis").show()
```
-### Adding A Catalog
+### Adding catalogs
-Iceberg has several catalog back-ends that can be used to track tables, like JDBC, Hive MetaStore and Glue.
-Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`. In this guide,
-we use JDBC, but you can follow these instructions to configure other catalog types. To learn more, check out
-the [Catalog](docs/latest/spark-configuration.md#catalogs) page in the Spark section.
+Apache Iceberg provides several catalog implementations to manage tables and enable SQL operations.
+Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`.
+You can configure different catalog types, such as JDBC, Hive Metastore, Glue, and REST, to manage Iceberg tables in Spark.
-This configuration creates a path-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
+This guide covers the configuration of two popular catalog types:
+
+* JDBC Catalog
+* REST Catalog
+
+To learn more, check out the [Catalog](docs/latest/spark-configuration.md#catalogs) page in the Spark section.
+
+#### Configuring JDBC Catalog
+
+The JDBC catalog stores Iceberg table metadata in a relational database.
+
+This configuration creates a JDBC-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
+
+The JDBC catalog uses file-based SQLite database as the backend.
=== "CLI"
```sh
- spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}\
+ spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
- --conf spark.sql.catalog.local.type=hadoop \
+ --conf spark.sql.catalog.local.type=jdbc \
+ --conf spark.sql.catalog.local.uri=jdbc:sqlite:$PWD/iceberg_catalog_db.sqlite \
--conf spark.sql.catalog.local.warehouse=$PWD/warehouse \
--conf spark.sql.defaultCatalog=local
```
=== "spark-defaults.conf"
```sh
- spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}
+ spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
spark.sql.catalog.local org.apache.iceberg.spark.SparkCatalog
- spark.sql.catalog.local.type hadoop
- spark.sql.catalog.local.warehouse $PWD/warehouse
+ spark.sql.catalog.local.type jdbc
+ spark.sql.catalog.local.uri jdbc:sqlite:iceberg_catalog_db.sqlite
+ spark.sql.catalog.local.warehouse warehouse
spark.sql.defaultCatalog local
```
!!! note
If your Iceberg catalog is not set as the default catalog, you will have to switch to it by executing `USE local;`
+#### Configuring REST Catalog
Review Comment:
I would be inclined to move this as the first option :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [docs] Replace examples of Hadoop catalog with JDBC catalog [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #11845:
URL: https://github.com/apache/iceberg/pull/11845#discussion_r1895040536
##########
docs/docs/spark-getting-started.md:
##########
Review Comment:
note, there are two "getting started" docs
this one and `site/docs/spark-quickstart.md`
##########
site/docs/spark-quickstart.md:
##########
@@ -26,7 +26,11 @@ highlight some powerful features. You can learn more about Iceberg's Spark runti
- [Writing Data to a Table](#writing-data-to-a-table)
- [Reading Data from a Table](#reading-data-from-a-table)
- [Adding A Catalog](#adding-a-catalog)
-- [Next Steps](#next-steps)
+ - [Configuring JDBC Catalog](#configuring-jdbc-catalog)
+ - [Configuring REST Catalog](#configuring-rest-catalog)
+- [Next steps](#next-steps)
+ - [Adding Iceberg to Spark](#adding-iceberg-to-spark)
+ - [Learn More](#learn-more)
Review Comment:
renders the subsection correctly
##########
site/docs/spark-quickstart.md:
##########
@@ -269,42 +273,104 @@ To read a table, simply use the Iceberg table's name.
### Adding A Catalog
-Iceberg has several catalog back-ends that can be used to track tables, like JDBC, Hive MetaStore and Glue.
-Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`. In this guide,
-we use JDBC, but you can follow these instructions to configure other catalog types. To learn more, check out
Review Comment:
weird that the guide already mention JDBC here, but the example is still hadoop
##########
site/docs/spark-quickstart.md:
##########
@@ -267,44 +271,109 @@ To read a table, simply use the Iceberg table's name.
df = spark.table("demo.nyc.taxis").show()
```
-### Adding A Catalog
+### Adding catalogs
-Iceberg has several catalog back-ends that can be used to track tables, like JDBC, Hive MetaStore and Glue.
-Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`. In this guide,
-we use JDBC, but you can follow these instructions to configure other catalog types. To learn more, check out
-the [Catalog](docs/latest/spark-configuration.md#catalogs) page in the Spark section.
+Apache Iceberg provides several catalog implementations to manage tables and enable SQL operations.
+Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`.
+You can configure different catalog types, such as JDBC, Hive Metastore, Glue, and REST, to manage Iceberg tables in Spark.
-This configuration creates a path-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
+This guide covers the configuration of two popular catalog types:
+
+* JDBC Catalog
+* REST Catalog
+
+To learn more, check out the [Catalog](docs/latest/spark-configuration.md#catalogs) page in the Spark section.
+
+#### Configuring JDBC Catalog
+
+The JDBC catalog stores Iceberg table metadata in a relational database.
+
+This configuration creates a JDBC-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
+
+The JDBC catalog uses file-based SQLite database as the backend.
=== "CLI"
```sh
- spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}\
+ spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
- --conf spark.sql.catalog.local.type=hadoop \
+ --conf spark.sql.catalog.local.type=jdbc \
+ --conf spark.sql.catalog.local.uri=jdbc:sqlite:$PWD/iceberg_catalog_db.sqlite \
--conf spark.sql.catalog.local.warehouse=$PWD/warehouse \
--conf spark.sql.defaultCatalog=local
```
=== "spark-defaults.conf"
```sh
- spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}
+ spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
spark.sql.catalog.local org.apache.iceberg.spark.SparkCatalog
- spark.sql.catalog.local.type hadoop
- spark.sql.catalog.local.warehouse $PWD/warehouse
Review Comment:
`$PWD` does not expand in `spark-defaults.conf`. keeping this here will create a folder named `$PWD`
##########
docs/docs/spark-getting-started.md:
##########
@@ -41,20 +41,26 @@ spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ iceb
### Adding catalogs
-Iceberg comes with [catalogs](spark-configuration.md#catalogs) that enable SQL commands to manage tables and load them by name. Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`.
+Apache Iceberg provides several [catalog](spark-configuration.md#catalogs) implementations to manage tables and enable SQL operations.
+Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`.
-This command creates a path-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog:
+This command creates a JDBC-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
+
+The JDBC catalog uses file-based SQLite database as the backend.
```sh
-spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}\
+spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
- --conf spark.sql.catalog.local.type=hadoop \
+ --conf spark.sql.catalog.local.type=jdbc \
+ --conf spark.sql.catalog.local.uri=jdbc:sqlite:$PWD/iceberg_catalog_db.sqlite \
--conf spark.sql.catalog.local.warehouse=$PWD/warehouse
```
+For example configuring a REST-based catalog, see [Configuring REST Catalog](/spark-quickstart#configuring-rest-catalog)
Review Comment:
instead of repeating here for configuring REST catalog, just link to `site/docs/spark-quickstart.md`. I double checked the link here locally
##########
docs/docs/spark-getting-started.md:
##########
@@ -41,20 +41,27 @@ spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ iceb
### Adding catalogs
-Iceberg comes with [catalogs](spark-configuration.md#catalogs) that enable SQL commands to manage tables and load them by name. Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`.
+Apache Iceberg provides several [catalog](spark-configuration.md#catalogs) implementations to manage tables and enable SQL operations.
+Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`.
-This command creates a path-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog:
+This command creates a JDBC-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
+
+The JDBC catalog uses file-based SQLite database as the backend.
```sh
-spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}\
+spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
- --conf spark.sql.catalog.local.type=hadoop \
- --conf spark.sql.catalog.local.warehouse=$PWD/warehouse
+ --conf spark.sql.catalog.local.type=jdbc \
+ --conf spark.sql.catalog.local.uri=jdbc:sqlite:$PWD/iceberg_catalog_db.sqlite \
+ --conf spark.sql.catalog.local.warehouse=$PWD/warehouse \
+ --conf spark.sql.defaultCatalog=local
Review Comment:
add `defaultCatalog` to match other pages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [docs] Replace examples of Hadoop catalog with JDBC & REST catalog [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11845:
URL: https://github.com/apache/iceberg/pull/11845#discussion_r1934425980
##########
docs/docs/spark-getting-started.md:
##########
@@ -39,22 +39,28 @@ spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ iceb
If you want to include Iceberg in your Spark installation, add the [`iceberg-spark-runtime-3.5_2.12` Jar](https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/{{ icebergVersion }}/iceberg-spark-runtime-3.5_2.12-{{ icebergVersion }}.jar) to Spark's `jars` folder.
-### Adding catalogs
+### Catalogs
-Iceberg comes with [catalogs](spark-configuration.md#catalogs) that enable SQL commands to manage tables and load them by name. Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`.
+Apache Iceberg uses [catalogs](https://iceberg.apache.org/concepts/catalog/) to track and manage table metadata and enable SQL operations. Iceberg provides several [catalog implementations](spark-configuration.md#catalogs), which can be configured using properties under `spark.sql.catalog.(catalog_name)`.
-This command creates a path-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog:
+This command creates a JDBC-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
Review Comment:
```suggestion
This command creates a JDBC-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in `spark_catalog` using the Hive connector.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [docs] Replace examples of Hadoop catalog with JDBC catalog [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #11845:
URL: https://github.com/apache/iceberg/pull/11845#discussion_r1913037140
##########
site/docs/spark-quickstart.md:
##########
@@ -267,44 +271,109 @@ To read a table, simply use the Iceberg table's name.
df = spark.table("demo.nyc.taxis").show()
```
-### Adding A Catalog
+### Adding catalogs
Review Comment:
I would even consider just calling this Catalogs:
```suggestion
### Catalogs
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [docs] Replace examples of Hadoop catalog with JDBC catalog [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #11845:
URL: https://github.com/apache/iceberg/pull/11845#discussion_r1913641156
##########
site/docs/spark-quickstart.md:
##########
@@ -267,44 +271,109 @@ To read a table, simply use the Iceberg table's name.
df = spark.table("demo.nyc.taxis").show()
```
-### Adding A Catalog
+### Adding catalogs
-Iceberg has several catalog back-ends that can be used to track tables, like JDBC, Hive MetaStore and Glue.
-Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`. In this guide,
-we use JDBC, but you can follow these instructions to configure other catalog types. To learn more, check out
-the [Catalog](docs/latest/spark-configuration.md#catalogs) page in the Spark section.
+Apache Iceberg provides several catalog implementations to manage tables and enable SQL operations.
+Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`.
+You can configure different catalog types, such as JDBC, Hive Metastore, Glue, and REST, to manage Iceberg tables in Spark.
-This configuration creates a path-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
+This guide covers the configuration of two popular catalog types:
+
+* JDBC Catalog
+* REST Catalog
+
+To learn more, check out the [Catalog](docs/latest/spark-configuration.md#catalogs) page in the Spark section.
+
+#### Configuring JDBC Catalog
+
+The JDBC catalog stores Iceberg table metadata in a relational database.
+
+This configuration creates a JDBC-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
+
+The JDBC catalog uses file-based SQLite database as the backend.
=== "CLI"
```sh
- spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}\
+ spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
- --conf spark.sql.catalog.local.type=hadoop \
+ --conf spark.sql.catalog.local.type=jdbc \
+ --conf spark.sql.catalog.local.uri=jdbc:sqlite:$PWD/iceberg_catalog_db.sqlite \
--conf spark.sql.catalog.local.warehouse=$PWD/warehouse \
--conf spark.sql.defaultCatalog=local
```
=== "spark-defaults.conf"
```sh
- spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}
+ spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
spark.sql.catalog.local org.apache.iceberg.spark.SparkCatalog
- spark.sql.catalog.local.type hadoop
- spark.sql.catalog.local.warehouse $PWD/warehouse
+ spark.sql.catalog.local.type jdbc
+ spark.sql.catalog.local.uri jdbc:sqlite:iceberg_catalog_db.sqlite
+ spark.sql.catalog.local.warehouse warehouse
spark.sql.defaultCatalog local
```
!!! note
If your Iceberg catalog is not set as the default catalog, you will have to switch to it by executing `USE local;`
+#### Configuring REST Catalog
Review Comment:
done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [docs] Replace examples of Hadoop catalog with JDBC catalog [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #11845:
URL: https://github.com/apache/iceberg/pull/11845#discussion_r1913038717
##########
site/docs/spark-quickstart.md:
##########
@@ -267,44 +271,109 @@ To read a table, simply use the Iceberg table's name.
df = spark.table("demo.nyc.taxis").show()
```
-### Adding A Catalog
+### Adding catalogs
-Iceberg has several catalog back-ends that can be used to track tables, like JDBC, Hive MetaStore and Glue.
-Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`. In this guide,
-we use JDBC, but you can follow these instructions to configure other catalog types. To learn more, check out
-the [Catalog](docs/latest/spark-configuration.md#catalogs) page in the Spark section.
+Apache Iceberg provides several catalog implementations to manage tables and enable SQL operations.
+Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`.
+You can configure different catalog types, such as JDBC, Hive Metastore, Glue, and REST, to manage Iceberg tables in Spark.
-This configuration creates a path-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
+This guide covers the configuration of two popular catalog types:
+
+* JDBC Catalog
+* REST Catalog
+
+To learn more, check out the [Catalog](docs/latest/spark-configuration.md#catalogs) page in the Spark section.
+
+#### Configuring JDBC Catalog
+
+The JDBC catalog stores Iceberg table metadata in a relational database.
+
+This configuration creates a JDBC-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
+
+The JDBC catalog uses file-based SQLite database as the backend.
=== "CLI"
```sh
- spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}\
+ spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
- --conf spark.sql.catalog.local.type=hadoop \
+ --conf spark.sql.catalog.local.type=jdbc \
+ --conf spark.sql.catalog.local.uri=jdbc:sqlite:$PWD/iceberg_catalog_db.sqlite \
--conf spark.sql.catalog.local.warehouse=$PWD/warehouse \
--conf spark.sql.defaultCatalog=local
```
=== "spark-defaults.conf"
```sh
- spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}
+ spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
spark.sql.catalog.local org.apache.iceberg.spark.SparkCatalog
- spark.sql.catalog.local.type hadoop
- spark.sql.catalog.local.warehouse $PWD/warehouse
+ spark.sql.catalog.local.type jdbc
+ spark.sql.catalog.local.uri jdbc:sqlite:iceberg_catalog_db.sqlite
+ spark.sql.catalog.local.warehouse warehouse
spark.sql.defaultCatalog local
```
!!! note
If your Iceberg catalog is not set as the default catalog, you will have to switch to it by executing `USE local;`
+#### Configuring REST Catalog
+
+The REST catalog provides a language-agnostic way to manage Iceberg tables through a RESTful service.
+
+This configuration creates a REST-based catalog named `rest` for tables under `s3://warehouse/` and adds support for Iceberg tables to Spark's built-in catalog.
+
+The REST catalog uses the `apache/iceberg-rest-fixture` docker container from the `docker-compose.yml` above as the backend service with MinIO for S3-compatible storage.
+
+=== "CLI"
+
+ ```sh
+ spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.apache.iceberg:iceberg-aws-bundle:1.7.1 \
+ --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
+ --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
+ --conf spark.sql.catalog.spark_catalog.type=hive \
+ --conf spark.sql.catalog.rest=org.apache.iceberg.spark.SparkCatalog \
+ --conf spark.sql.catalog.rest.type=rest \
+ --conf spark.sql.catalog.rest.uri=http://localhost:8181 \
+ --conf spark.sql.catalog.rest.warehouse=s3://warehouse/ \
+ --conf spark.sql.catalog.rest.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
+ --conf spark.sql.catalog.rest.s3.endpoint=http://localhost:9000 \
+ --conf spark.sql.catalog.rest.s3.path-style-access=true \
+ --conf spark.sql.catalog.rest.s3.access-key-id=admin \
+ --conf spark.sql.catalog.rest.s3.secret-access-key=password \
+ --conf spark.sql.catalog.rest.client.region=us-east-1 \
+ --conf spark.sql.defaultCatalog=rest
+ ```
+
+=== "spark-defaults.conf"
+
+ ```sh
+ spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.apache.iceberg:iceberg-aws-bundle:1.7.1
+ spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
+ spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
+ spark.sql.catalog.spark_catalog.type hive
+ spark.sql.catalog.rest org.apache.iceberg.spark.SparkCatalog
+ spark.sql.catalog.rest.type rest
+ spark.sql.catalog.rest.uri http://localhost:8181
+ spark.sql.catalog.rest.warehouse s3://warehouse/
+ spark.sql.catalog.rest.io-impl org.apache.iceberg.aws.s3.S3FileIO
+ spark.sql.catalog.rest.s3.endpoint http://localhost:9000
+ spark.sql.catalog.rest.s3.path-style-access true
+ spark.sql.catalog.rest.s3.access-key-id admin
+ spark.sql.catalog.rest.s3.secret-access-key password
+ spark.sql.catalog.rest.client.region us-east-1
+ spark.sql.defaultCatalog rest
+ ```
+
+!!! note
+ If your Iceberg catalog is not set as the default catalog, you will have to switch to it by executing `USE rest;`
Review Comment:
```suggestion
If your Iceberg catalog is not set as the default catalog using `spark.sql.defaultCatalog`, you will have to switch to it by executing `USE rest;`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [docs] Replace examples of Hadoop catalog with JDBC & REST catalog [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #11845:
URL: https://github.com/apache/iceberg/pull/11845#discussion_r1935822049
##########
docs/docs/spark-getting-started.md:
##########
@@ -39,22 +39,28 @@ spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ iceb
If you want to include Iceberg in your Spark installation, add the [`iceberg-spark-runtime-3.5_2.12` Jar](https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/{{ icebergVersion }}/iceberg-spark-runtime-3.5_2.12-{{ icebergVersion }}.jar) to Spark's `jars` folder.
-### Adding catalogs
+### Catalogs
-Iceberg comes with [catalogs](spark-configuration.md#catalogs) that enable SQL commands to manage tables and load them by name. Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`.
+Apache Iceberg uses [catalogs](https://iceberg.apache.org/concepts/catalog/) to track and manage table metadata and enable SQL operations. Iceberg provides several [catalog implementations](spark-configuration.md#catalogs), which can be configured using properties under `spark.sql.catalog.(catalog_name)`.
-This command creates a path-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog:
+This command creates a JDBC-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog.
Review Comment:
maybe something like this,
```suggestion
This command creates a JDBC-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog (`spark_catalog`) using the Hive connector.
```
but I'm also incline not to add this. I feel like this is too detailed for a "getting started" page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [docs] Replace examples of Hadoop catalog with JDBC & REST catalog [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11845:
URL: https://github.com/apache/iceberg/pull/11845#issuecomment-2692483140
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org