You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/06/05 03:32:05 UTC
[GitHub] [iceberg] hzfanxinxin opened a new pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
hzfanxinxin opened a new pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095
This PR propose to add HadoopCatalog example to create/load tables in the Iceberg User docs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095#discussion_r440488580
##########
File path: site/docs/api-quickstart.md
##########
@@ -48,6 +48,36 @@ logsDF.write
The logs [schema](#create-a-schema) and [partition spec](#create-a-partition-spec) are created below.
+### Using a Hadoop catalog
+
+The Hadoop catalog doesn't need to connects to a Hive MetaStore. To get a Hadoop catalog see:
+
+```scala
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+
+val conf = new Configuration();
+val warehousePath = "hdfs://warehouse_path";
+val catalog = new HadoopCatalog(conf, warehousePath);
+```
+
+Like Hive catalog, Hadoop catalog implements the interface `Catalog`. So it also contains methods for working with tables, like createTable, loadTable, renameTable, and dropTable.
+
+This example create a table with Hadoop catalog:
+
+```scala
+val name = TableIdentifier.of("logging", "logs")
+val table = catalog.createTable(name, schema, spec)
+
+// write into the new logs table with Spark 2.4
+logsDF.write
+ .format("iceberg")
+ .mode("append")
+ .save("hdfs://warehouse_path/logging/logs")
Review comment:
This seems like something we might want to solve in configuration instead of hard-coding support for `HiveCatalog`. This problem might go away when we get a better API in Spark 3.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095#discussion_r441721593
##########
File path: site/docs/api-quickstart.md
##########
@@ -48,6 +48,36 @@ logsDF.write
The logs [schema](#create-a-schema) and [partition spec](#create-a-partition-spec) are created below.
+### Using a Hadoop catalog
+
+A Hadoop catalog doesn't need to connect to a Hive MetaStore, but can only be used with HDFS or similar file systems that support atomic rename. To get a Hadoop catalog see:
+
+```scala
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+
+val conf = new Configuration();
+val warehousePath = "hdfs://host:8020/warehouse_path";
+val catalog = new HadoopCatalog(conf, warehousePath);
+```
+
+Like Hive catalog, Hadoop catalog implements the interface `Catalog`. So it also contains methods for working with tables, like createTable, loadTable, and dropTable.
+
+This example create a table with Hadoop catalog:
+
+```scala
+val name = TableIdentifier.of("logging", "logs")
+val table = catalog.createTable(name, schema, spec)
+
+// write into the new logs table with Spark 2.4
+logsDF.write
+ .format("iceberg")
+ .mode("append")
+ .save("hdfs://warehouse_path/logging/logs")
Review comment:
This location also needs to be fixed to avoid using `warehouse_path` as the authority section of the URI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095#discussion_r440488024
##########
File path: site/docs/api-quickstart.md
##########
@@ -48,6 +48,36 @@ logsDF.write
The logs [schema](#create-a-schema) and [partition spec](#create-a-partition-spec) are created below.
+### Using a Hadoop catalog
+
+The Hadoop catalog doesn't need to connects to a Hive MetaStore. To get a Hadoop catalog see:
+
+```scala
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+
+val conf = new Configuration();
+val warehousePath = "hdfs://warehouse_path";
Review comment:
No, then `warehouse` is the authority section. You should change it to something like `hdfs://host:8020/warehouse/path/`
##########
File path: site/docs/api-quickstart.md
##########
@@ -48,6 +48,36 @@ logsDF.write
The logs [schema](#create-a-schema) and [partition spec](#create-a-partition-spec) are created below.
+### Using a Hadoop catalog
+
+The Hadoop catalog doesn't need to connects to a Hive MetaStore. To get a Hadoop catalog see:
+
+```scala
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+
+val conf = new Configuration();
+val warehousePath = "hdfs://warehouse_path";
+val catalog = new HadoopCatalog(conf, warehousePath);
+```
+
+Like Hive catalog, Hadoop catalog implements the interface `Catalog`. So it also contains methods for working with tables, like createTable, loadTable, renameTable, and dropTable.
Review comment:
Yes, please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hzfanxinxin commented on a change in pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
Posted by GitBox <gi...@apache.org>.
hzfanxinxin commented on a change in pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095#discussion_r438488723
##########
File path: site/docs/api-quickstart.md
##########
@@ -48,6 +48,36 @@ logsDF.write
The logs [schema](#create-a-schema) and [partition spec](#create-a-partition-spec) are created below.
+### Using a Hadoop catalog
+
+The Hadoop catalog doesn't need to connects to a Hive MetaStore. To get a Hadoop catalog see:
Review comment:
I think so.
##########
File path: site/docs/api-quickstart.md
##########
@@ -48,6 +48,36 @@ logsDF.write
The logs [schema](#create-a-schema) and [partition spec](#create-a-partition-spec) are created below.
+### Using a Hadoop catalog
+
+The Hadoop catalog doesn't need to connects to a Hive MetaStore. To get a Hadoop catalog see:
+
+```scala
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+
+val conf = new Configuration();
+val warehousePath = "hdfs://warehouse_path";
+val catalog = new HadoopCatalog(conf, warehousePath);
+```
+
+Like Hive catalog, Hadoop catalog implements the interface `Catalog`. So it also contains methods for working with tables, like createTable, loadTable, renameTable, and dropTable.
Review comment:
Yes, you are right. So should i remove renameTable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095#discussion_r438326656
##########
File path: site/docs/api-quickstart.md
##########
@@ -48,6 +48,36 @@ logsDF.write
The logs [schema](#create-a-schema) and [partition spec](#create-a-partition-spec) are created below.
+### Using a Hadoop catalog
+
+The Hadoop catalog doesn't need to connects to a Hive MetaStore. To get a Hadoop catalog see:
+
+```scala
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+
+val conf = new Configuration();
+val warehousePath = "hdfs://warehouse_path";
+val catalog = new HadoopCatalog(conf, warehousePath);
+```
+
+Like Hive catalog, Hadoop catalog implements the interface `Catalog`. So it also contains methods for working with tables, like createTable, loadTable, renameTable, and dropTable.
+
+This example create a table with Hadoop catalog:
+
+```scala
+val name = TableIdentifier.of("logging", "logs")
+val table = catalog.createTable(name, schema, spec)
+
+// write into the new logs table with Spark 2.4
+logsDF.write
+ .format("iceberg")
+ .mode("append")
+ .save("hdfs://warehouse_path/logging/logs")
Review comment:
This is strange because it actually loads the table using `HadoopTables` and not the `HadoopCatalog`. The URI passed here must match the one created by the catalog.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue edited a comment on pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
Posted by GitBox <gi...@apache.org>.
rdblue edited a comment on pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095#issuecomment-645527818
I did some minor editing and fixed the read path. Overall, looks good so I'll merge this.
Thanks for helping make the docs better, @hzfanxinxin!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hzfanxinxin commented on a change in pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
Posted by GitBox <gi...@apache.org>.
hzfanxinxin commented on a change in pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095#discussion_r438493845
##########
File path: site/docs/api-quickstart.md
##########
@@ -48,6 +48,36 @@ logsDF.write
The logs [schema](#create-a-schema) and [partition spec](#create-a-partition-spec) are created below.
+### Using a Hadoop catalog
+
+The Hadoop catalog doesn't need to connects to a Hive MetaStore. To get a Hadoop catalog see:
+
+```scala
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+
+val conf = new Configuration();
+val warehousePath = "hdfs://warehouse_path";
+val catalog = new HadoopCatalog(conf, warehousePath);
+```
+
+Like Hive catalog, Hadoop catalog implements the interface `Catalog`. So it also contains methods for working with tables, like createTable, loadTable, renameTable, and dropTable.
+
+This example create a table with Hadoop catalog:
+
+```scala
+val name = TableIdentifier.of("logging", "logs")
+val table = catalog.createTable(name, schema, spec)
+
+// write into the new logs table with Spark 2.4
+logsDF.write
+ .format("iceberg")
+ .mode("append")
+ .save("hdfs://warehouse_path/logging/logs")
Review comment:
Yes, this is also a problem that bothers me. I look the method `findTable` in class IcebergSource.java, it just support `HadoopTables` and `HiveCatalog`. So i think if we can add a option like `.option("catalog","HadoopCatalog")` to the DataSourceOptions, and the default value of catalog option is HiveCatalog. what's your opinion on this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hzfanxinxin commented on pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
Posted by GitBox <gi...@apache.org>.
hzfanxinxin commented on pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095#issuecomment-645218608
@rdblue would you please help to review again, thanks. I've addressed all the comments except one about `HadoopTables` loads the tables which created by `HadoopCatalog`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095#issuecomment-645527818
I did some minor editing and fixed the read path. Overall, looks good so I'll merge this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095#discussion_r438326200
##########
File path: site/docs/api-quickstart.md
##########
@@ -48,6 +48,36 @@ logsDF.write
The logs [schema](#create-a-schema) and [partition spec](#create-a-partition-spec) are created below.
+### Using a Hadoop catalog
+
+The Hadoop catalog doesn't need to connects to a Hive MetaStore. To get a Hadoop catalog see:
+
+```scala
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+
+val conf = new Configuration();
+val warehousePath = "hdfs://warehouse_path";
+val catalog = new HadoopCatalog(conf, warehousePath);
+```
+
+Like Hive catalog, Hadoop catalog implements the interface `Catalog`. So it also contains methods for working with tables, like createTable, loadTable, renameTable, and dropTable.
Review comment:
I don't think rename is implemented, is it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hzfanxinxin commented on a change in pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
Posted by GitBox <gi...@apache.org>.
hzfanxinxin commented on a change in pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095#discussion_r438490978
##########
File path: site/docs/api-quickstart.md
##########
@@ -48,6 +48,36 @@ logsDF.write
The logs [schema](#create-a-schema) and [partition spec](#create-a-partition-spec) are created below.
+### Using a Hadoop catalog
+
+The Hadoop catalog doesn't need to connects to a Hive MetaStore. To get a Hadoop catalog see:
+
+```scala
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+
+val conf = new Configuration();
+val warehousePath = "hdfs://warehouse_path";
Review comment:
`hdfs://warehouse/path`, is it ok?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095#discussion_r438325318
##########
File path: site/docs/api-quickstart.md
##########
@@ -48,6 +48,36 @@ logsDF.write
The logs [schema](#create-a-schema) and [partition spec](#create-a-partition-spec) are created below.
+### Using a Hadoop catalog
+
+The Hadoop catalog doesn't need to connects to a Hive MetaStore. To get a Hadoop catalog see:
Review comment:
This needs to note the disadvantages of using HadoopCatalog so that users are informed about the problems. Specifically, this can only be used with HDFS. Can you add that to the first sentence, like this?
> A Hadoop catalog doesn't need to connect to a Hive MetaStore, but can only be used with HDFS or similar file systems that support atomic rename.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1095: Add HadoopCatalog example to create/load tables in the Iceberg User docs
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1095:
URL: https://github.com/apache/iceberg/pull/1095#discussion_r438326017
##########
File path: site/docs/api-quickstart.md
##########
@@ -48,6 +48,36 @@ logsDF.write
The logs [schema](#create-a-schema) and [partition spec](#create-a-partition-spec) are created below.
+### Using a Hadoop catalog
+
+The Hadoop catalog doesn't need to connects to a Hive MetaStore. To get a Hadoop catalog see:
+
+```scala
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+
+val conf = new Configuration();
+val warehousePath = "hdfs://warehouse_path";
Review comment:
This doesn't use the right URI format, `schema://authority/path`. This URI would use `warehouse_path` for the HDFS host in the authority section.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org