You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/10/16 16:17:14 UTC
[GitHub] [iceberg] mrn-aglic opened a new issue, #5993: Quick start docker-compose demo doesn't work
mrn-aglic opened a new issue, #5993:
URL: https://github.com/apache/iceberg/issues/5993
### Apache Iceberg version
0.14.1 (latest release)
### Query engine
Spark
### Please describe the bug ๐
Here is the docker-compose file that I tried to use:
```
services:
postgres:
image: postgres:13.4-bullseye
container_name: iceberg_postgres
environment:
- POSTGRES_USER=admin
- POSTGRES_PASSWORD=password
- POSTGRES_DB=demo_catalog
volumes:
- ./postgres/data:/var/lib/postgresql/data
spark-iceberg:
image: tabulario/spark-iceberg
depends_on:
- postgres
container_name: spark-iceberg
environment:
- SPARK_HOME=/opt/spark
- PYSPARK_PYTON=/usr/bin/python3.9
- PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/spark/bin
volumes:
- ./warehouse:/home/iceberg/warehouse
- ./notebooks:/home/iceberg/notebooks/notebooks
ports:
- 8888:8888
- 8080:8080
- 18080:18080
```
There are 2 issues I came across:
1. localhost:8080 and localhost:18080 are not accessible from my web browser (Firefox and Safari)
2. Trying to run the example notebook Iceberg - Getting started, I can run the "spark" cell (the first cell), but the next one throws an error:
```
Py4JJavaError Traceback (most recent call last)
Cell In [4], line 1
----> 1 get_ipython().run_cell_magic('sql', '', '\nDROP TABLE IF EXISTS nyc.taxis\n')
File /usr/local/lib/python3.9/site-packages/IPython/core/interactiveshell.py:2362, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
2360 with self.builtin_trap:
2361 args = (magic_arg_s, cell)
-> 2362 result = fn(*args, **kwargs)
2363 return result
File ~/.ipython/profile_default/startup/00-prettytables.py:61, in sql(line, cell)
59 return _to_table(df, num_rows=args.limit)
60 else:
---> 61 return _to_table(spark.sql(cell))
File /opt/spark/python/pyspark/sql/session.py:1034, in SparkSession.sql(self, sqlQuery, **kwargs)
1032 sqlQuery = formatter.format(sqlQuery, **kwargs)
1033 try:
-> 1034 return DataFrame(self._jsparkSession.sql(sqlQuery), self)
1035 finally:
1036 if len(kwargs) > 0:
File /opt/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py:1321, in JavaMember.__call__(self, *args)
1315 command = proto.CALL_COMMAND_NAME +\
1316 self.command_header +\
1317 args_command +\
1318 proto.END_COMMAND_PART
1320 answer = self.gateway_client.send_command(command)
-> 1321 return_value = get_return_value(
1322 answer, self.gateway_client, self.target_id, self.name)
1324 for temp_arg in temp_args:
1325 temp_arg._detach()
File /opt/spark/python/pyspark/sql/utils.py:190, in capture_sql_exception.<locals>.deco(*a, **kw)
188 def deco(*a: Any, **kw: Any) -> Any:
189 try:
--> 190 return f(*a, **kw)
191 except Py4JJavaError as e:
192 converted = convert_exception(e.java_exception)
File /opt/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name)
324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
325 if answer[1] == REFERENCE_TYPE:
--> 326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
328 format(target_id, ".", name), value)
329 else:
330 raise Py4JError(
331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
332 format(target_id, ".", name, value))
Py4JJavaError: An error occurred while calling o36.sql.
: org.apache.iceberg.exceptions.RESTException: Error occurred while processing GET request
```
The errors are probably related.
Sorry if wrongly reported as bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] fortarch commented on issue #5993: Quick start docker-compose demo doesn't work
Posted by GitBox <gi...@apache.org>.
fortarch commented on issue #5993:
URL: https://github.com/apache/iceberg/issues/5993#issuecomment-1319451204
> version: "3"
>
> services:
> spark-iceberg:
> image: tabulario/spark-iceberg
> container_name: spark-iceberg
> build: spark/
> depends_on:
> - rest
> - minio
> volumes:
> - ./warehouse:/home/iceberg/warehouse
> - ./notebooks:/home/iceberg/notebooks/notebooks
> environment:
> - AWS_ACCESS_KEY_ID=admin
> - AWS_SECRET_ACCESS_KEY=password
> - AWS_REGION=us-east-1
> ports:
> - 8888:8888
> - 8080:8080
> links:
> - rest:rest
> - minio:minio
> rest:
> image: tabulario/iceberg-rest:0.1.0
> ports:
> - 8181:8181
> environment:
> - AWS_ACCESS_KEY_ID=admin
> - AWS_SECRET_ACCESS_KEY=password
> - AWS_REGION=us-east-1
> - CATALOG_WAREHOUSE=s3a://warehouse/wh/
> - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
> - CATALOG_S3_ENDPOINT=http://minio:9000
> minio:
> image: minio/minio
> container_name: minio
> environment:
> - MINIO_ROOT_USER=admin
> - MINIO_ROOT_PASSWORD=password
> ports:
> - 9001:9001
> - 9000:9000
> command: ["server", "/data", "--console-address", ":9001"]
> mc:
> depends_on:
> - minio
> image: minio/mc
> container_name: mc
> environment:
> - AWS_ACCESS_KEY_ID=admin
> - AWS_SECRET_ACCESS_KEY=password
> - AWS_REGION=us-east-1
> entrypoint: >
> /bin/sh -c "
> until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
> /usr/bin/mc rm -r --force minio/warehouse;
> /usr/bin/mc mb minio/warehouse;
> /usr/bin/mc policy set public minio/warehouse;
> exit 0;
> "
It works. Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #5993: Quick start docker-compose demo doesn't work
Posted by GitBox <gi...@apache.org>.
Fokko commented on issue #5993:
URL: https://github.com/apache/iceberg/issues/5993#issuecomment-1318904274
The latest version of the image has been pushed. Please run `docker-compose pull` to fetch the latest images. Let me know if this works ๐๐ป
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] fortarch commented on issue #5993: Quick start docker-compose demo doesn't work
Posted by GitBox <gi...@apache.org>.
fortarch commented on issue #5993:
URL: https://github.com/apache/iceberg/issues/5993#issuecomment-1318234553
get exception, please resolve that.
`
Caused by: java.net.UnknownHostException: rest: Name or service not known
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #5993: Quick start docker-compose demo doesn't work
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #5993:
URL: https://github.com/apache/iceberg/issues/5993#issuecomment-1282412345
@samredai is this something you could have a look at please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhangjiuyang1993 commented on issue #5993: Quick start docker-compose demo doesn't work
Posted by "zhangjiuyang1993 (via GitHub)" <gi...@apache.org>.
zhangjiuyang1993 commented on issue #5993:
URL: https://github.com/apache/iceberg/issues/5993#issuecomment-1409640982
Sorry if wrongly reported as bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra closed issue #5993: Quick start docker-compose demo doesn't work
Posted by GitBox <gi...@apache.org>.
nastra closed issue #5993: Quick start docker-compose demo doesn't work
URL: https://github.com/apache/iceberg/issues/5993
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhangjiuyang1993 commented on issue #5993: Quick start docker-compose demo doesn't work
Posted by "zhangjiuyang1993 (via GitHub)" <gi...@apache.org>.
zhangjiuyang1993 commented on issue #5993:
URL: https://github.com/apache/iceberg/issues/5993#issuecomment-1409624924
@nastra
According to the configuration above, I got the following error๏ผ
23/01/31 01:43:58 WARN RESTSessionCatalog: Failed to report metrics to REST endpoint for table nyc.taxis
org.apache.iceberg.exceptions.BadRequestException: Malformed request: No route for request: POST v1/namespaces/nyc/tables/taxis/metrics
at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:152)
at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:135)
at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:150)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:224)
at org.apache.iceberg.rest.HTTPClient.post(HTTPClient.java:269)
at org.apache.iceberg.rest.RESTClient.post(RESTClient.java:112)
at org.apache.iceberg.rest.RESTSessionCatalog.reportMetrics(RESTSessionCatalog.java:321)
at org.apache.iceberg.rest.RESTSessionCatalog.lambda$loadTable$2(RESTSessionCatalog.java:307)
at org.apache.iceberg.BaseTableScan.lambda$planFiles$0(BaseTableScan.java:168)
at org.apache.iceberg.io.CloseableIterable$3.close(CloseableIterable.java:95)
at org.apache.iceberg.spark.source.SparkBatchQueryScan.files(SparkBatchQueryScan.java:125)
at org.apache.iceberg.spark.source.SparkBatchQueryScan.tasks(SparkBatchQueryScan.java:138)
at org.apache.iceberg.spark.source.SparkScan.toBatch(SparkScan.java:111)
at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.batch$lzycompute(BatchScanExec.scala:42)
at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.batch(BatchScanExec.scala:42)
at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.inputPartitions$lzycompute(BatchScanExec.scala:54)
at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.inputPartitions(BatchScanExec.scala:54)
at org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanExecBase.supportsColumnar(DataSourceV2ScanExecBase.scala:142)
at org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanExecBase.supportsColumnar$(DataSourceV2ScanExecBase.scala:141)
at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.supportsColumnar(BatchScanExec.scala:36)
at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Strategy.apply(DataSourceV2Strategy.scala:143)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$1(QueryPlanner.scala:63)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:491)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)
at org.apache.spark.sql.execution.SparkStrategies.plan(SparkStrategies.scala:69)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$3(QueryPlanner.scala:78)
at scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:196)
at scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:194)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:199)
at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:192)
at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1431)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$2(QueryPlanner.scala:75)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)
at org.apache.spark.sql.execution.SparkStrategies.plan(SparkStrategies.scala:69)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$3(QueryPlanner.scala:78)
at scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:196)
at scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:194)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:199)
at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:192)
at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1431)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$2(QueryPlanner.scala:75)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)
at org.apache.spark.sql.execution.SparkStrategies.plan(SparkStrategies.scala:69)
at org.apache.spark.sql.execution.QueryExecution$.createSparkPlan(QueryExecution.scala:459)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$sparkPlan$1(QueryExecution.scala:145)
at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:185)
at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:510)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:185)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:184)
at org.apache.spark.sql.execution.QueryExecution.sparkPlan$lzycompute(QueryExecution.scala:145)
at org.apache.spark.sql.execution.QueryExecution.sparkPlan(QueryExecution.scala:138)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executedPlan$1(QueryExecution.scala:158)
at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:185)
at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:510)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:185)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:184)
at org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:158)
at org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:151)
at org.apache.spark.sql.execution.QueryExecution.simpleString(QueryExecution.scala:204)
at org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$explainString(QueryExecution.scala:249)
at org.apache.spark.sql.execution.QueryExecution.explainString(QueryExecution.scala:218)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3856)
at org.apache.spark.sql.Dataset.collectToPython(Dataset.scala:3685)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
How to solve this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #5993: Quick start docker-compose demo doesn't work
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #5993:
URL: https://github.com/apache/iceberg/issues/5993#issuecomment-1319648430
Closing this as it's fixed by https://github.com/apache/iceberg-docs/pull/178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] adhithyan-vijayakumar commented on issue #5993: Quick start docker-compose demo doesn't work
Posted by GitBox <gi...@apache.org>.
adhithyan-vijayakumar commented on issue #5993:
URL: https://github.com/apache/iceberg/issues/5993#issuecomment-1280322455
I am also getting this on following the quickstart.
In addition to the above, I found the following line in traceback
```
Caused by: java.net.UnknownHostException: rest: Name or service not known
```
Docker is not resolving the rest address used to create urls. Iceberg team please help to resolve at the earliest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] gar3thjon3s commented on issue #5993: Quick start docker-compose demo doesn't work
Posted by GitBox <gi...@apache.org>.
gar3thjon3s commented on issue #5993:
URL: https://github.com/apache/iceberg/issues/5993#issuecomment-1284518979
It's a typo in the docker-compose.yml file - I had the same problem:
replace `PYSPARK_PYTON` -> `PYSPARK_PYTHON`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #5993: Quick start docker-compose demo doesn't work
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #5993:
URL: https://github.com/apache/iceberg/issues/5993#issuecomment-1318267420
This should be fixed by https://github.com/apache/iceberg-docs/pull/178. Until the docs have been updated on the [website](https://iceberg.apache.org/spark-quickstart/), you could use the following `docker-compose.yml`:
```
version: "3"
services:
spark-iceberg:
image: tabulario/spark-iceberg
container_name: spark-iceberg
build: spark/
depends_on:
- rest
- minio
volumes:
- ./warehouse:/home/iceberg/warehouse
- ./notebooks:/home/iceberg/notebooks/notebooks
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
- AWS_REGION=us-east-1
ports:
- 8888:8888
- 8080:8080
links:
- rest:rest
- minio:minio
rest:
image: tabulario/iceberg-rest:0.1.0
ports:
- 8181:8181
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
- AWS_REGION=us-east-1
- CATALOG_WAREHOUSE=s3a://warehouse/wh/
- CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
- CATALOG_S3_ENDPOINT=http://minio:9000
minio:
image: minio/minio
container_name: minio
environment:
- MINIO_ROOT_USER=admin
- MINIO_ROOT_PASSWORD=password
ports:
- 9001:9001
- 9000:9000
command: ["server", "/data", "--console-address", ":9001"]
mc:
depends_on:
- minio
image: minio/mc
container_name: mc
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
- AWS_REGION=us-east-1
entrypoint: >
/bin/sh -c "
until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
/usr/bin/mc rm -r --force minio/warehouse;
/usr/bin/mc mb minio/warehouse;
/usr/bin/mc policy set public minio/warehouse;
exit 0;
"
```
@fortarch please let me know if that works for you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on issue #5993: Quick start docker-compose demo doesn't work
Posted by GitBox <gi...@apache.org>.
samredai commented on issue #5993:
URL: https://github.com/apache/iceberg/issues/5993#issuecomment-1282610367
The latest image has been recently updated to use the REST catalog. This requires some slight changes to the docker compose which we've updated [here](https://github.com/tabular-io/docker-spark-iceberg/blob/main/docker-compose.yml) in the repo.
If you want to get back to the old setup that uses a `JDBCCatalog`, you can simply add a tag to the image name in your current docker compose setup: `tabulario/spark-iceberg:3.3.0_0.14.1`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #5993: Quick start docker-compose demo doesn't work
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #5993:
URL: https://github.com/apache/iceberg/issues/5993#issuecomment-1410476388
@zhangjiuyang1993 are you using `tabulario/iceberg-rest:0.1.0`? If so, you need to use the latest one with Iceberg 1.1.0 support, which is `tabulario/iceberg-rest:0.2.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org