You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@xtable.apache.org by "daragu (via GitHub)" <gi...@apache.org> on 2024/05/13 08:07:33 UTC
[PR] [Demo] Make start_demo script easier to call [incubator-xtable]
daragu opened a new pull request, #437:
URL: https://github.com/apache/incubator-xtable/pull/437
## *Important Read*
Fixed #436.
## What is the purpose of the pull request
We can only cd to the demo directory to execute the script currently, this purpose of this issue is to improve usability.
## Brief change log
Update `start_demo.sh` script.
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HudiSchemaEvolutionTest to verify the change.*
- *Manually verified the change by running a job locally.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown merged PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "xtable-bot (via GitHub)" <gi...@apache.org>.
xtable-bot commented on PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#issuecomment-2128828621
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=975",
"triggerID" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "2121345418",
"triggerType" : "MANUAL"
}, {
"hash" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=994",
"triggerID" : "2121345418",
"triggerType" : "MANUAL"
}, {
"hash" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=994",
"triggerID" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed Azure: [PENDING](https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=994)
<details>
<summary>Bot commands</summary>
@xtable-bot supports the following commands:
- `@xtable-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "xtable-bot (via GitHub)" <gi...@apache.org>.
xtable-bot commented on PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#issuecomment-2132476844
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=975",
"triggerID" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"triggerType" : "PUSH"
}, {
"hash" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=994",
"triggerID" : "2121345418",
"triggerType" : "MANUAL"
}, {
"hash" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=994",
"triggerID" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "2121345418",
"triggerType" : "MANUAL"
}, {
"hash" : "ebb4b0eec70bf4f2f6ef7d72aa5630b31fda20c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ebb4b0eec70bf4f2f6ef7d72aa5630b31fda20c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed Azure: [SUCCESS](https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=994)
* ebb4b0eec70bf4f2f6ef7d72aa5630b31fda20c9 UNKNOWN
<details>
<summary>Bot commands</summary>
@xtable-bot supports the following commands:
- `@xtable-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "xtable-bot (via GitHub)" <gi...@apache.org>.
xtable-bot commented on PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#issuecomment-2132477608
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=975",
"triggerID" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"triggerType" : "PUSH"
}, {
"hash" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=994",
"triggerID" : "2121345418",
"triggerType" : "MANUAL"
}, {
"hash" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=994",
"triggerID" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "2121345418",
"triggerType" : "MANUAL"
}, {
"hash" : "ebb4b0eec70bf4f2f6ef7d72aa5630b31fda20c9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=1001",
"triggerID" : "ebb4b0eec70bf4f2f6ef7d72aa5630b31fda20c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed Azure: [SUCCESS](https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=994)
* ebb4b0eec70bf4f2f6ef7d72aa5630b31fda20c9 Azure: [PENDING](https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=1001)
<details>
<summary>Bot commands</summary>
@xtable-bot supports the following commands:
- `@xtable-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "xtable-bot (via GitHub)" <gi...@apache.org>.
xtable-bot commented on PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#issuecomment-2113353543
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* becb98bee4e2fd133612ead830bc2ea2c98960de UNKNOWN
<details>
<summary>Bot commands</summary>
@xtable-bot supports the following commands:
- `@xtable-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "sagarlakshmipathy (via GitHub)" <gi...@apache.org>.
sagarlakshmipathy commented on code in PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#discussion_r1603757557
##########
website/docs/how-to.md:
##########
@@ -105,7 +105,7 @@ from pyspark.sql.types import *
# initialize the bucket
table_name = "people"
-local_base_path = "/tmp/hudi-dataset"
+local_base_path = "file:/tmp/hudi-dataset"
Review Comment:
@daragu do you have the exception?
I just tried copy pasting the entire code on my local and I didn't see an error/exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "daragu (via GitHub)" <gi...@apache.org>.
daragu commented on code in PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#discussion_r1604434129
##########
website/docs/how-to.md:
##########
@@ -105,7 +105,7 @@ from pyspark.sql.types import *
# initialize the bucket
table_name = "people"
-local_base_path = "/tmp/hudi-dataset"
+local_base_path = "file:/tmp/hudi-dataset"
Review Comment:
hi @sagarlakshmipathy, here is the exception.
```
[root@djump /work/xtable/spark-3.2.2-bin-hadoop3.2]# bin/pyspark --packages org.apache.hudi:hudi-spark3.2-bundle_2.12:0.14.0 --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension"
Python 3.9.9 (main, Feb 28 2024, 09:09:14)
[GCC 10.3.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
:: loading settings :: url = jar:file:/work/xtable/spark-3.2.2-bin-hadoop3.2/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
org.apache.hudi#hudi-spark3.2-bundle_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-9a61e9af-3f77-4da3-8f02-4742c83356c3;1.0
confs: [default]
found org.apache.hudi#hudi-spark3.2-bundle_2.12;0.14.0 in central
:: resolution report :: resolve 83ms :: artifacts dl 5ms
:: modules in use:
org.apache.hudi#hudi-spark3.2-bundle_2.12;0.14.0 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 1 | 0 | 0 | 0 || 1 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-9a61e9af-3f77-4da3-8f02-4742c83356c3
confs: [default]
0 artifacts copied, 1 already retrieved (0kB/3ms)
2024-05-17 14:51:08,478 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.2.2
/_/
Using Python version 3.9.9 (main, Feb 28 2024 09:09:14)
Spark context Web UI available at http://djump:4040
Spark context available as 'sc' (master = local[*], app id = local-1715928669676).
SparkSession available as 'spark'.
>>> from pyspark.sql.types import *
>>>
>>> # initialize the bucket
>>> table_name = "people"
>>> local_base_path = "/tmp/hudi-dataset"
>>>
>>> records = [
... (1, 'John', 25, 'NYC', '2023-09-28 00:00:00'),
... (2, 'Emily', 30, 'SFO', '2023-09-28 00:00:00'),
... (3, 'Michael', 35, 'ORD', '2023-09-28 00:00:00'),
... (4, 'Andrew', 40, 'NYC', '2023-10-28 00:00:00'),
... (5, 'Bob', 28, 'SEA', '2023-09-23 00:00:00'),
... (6, 'Charlie', 31, 'DFW', '2023-08-29 00:00:00')
... ]
>>>
>>> schema = StructType([
... StructField("id", IntegerType(), True),
... StructField("name", StringType(), True),
... StructField("age", IntegerType(), True),
... StructField("city", StringType(), True),
... StructField("create_ts", StringType(), True)
... ])
>>>
>>> df = spark.createDataFrame(records, schema)
>>>
>>> hudi_options = {
... 'hoodie.table.name': table_name,
... 'hoodie.datasource.write.partitionpath.field': 'city',
... 'hoodie.datasource.write.hive_style_partitioning': 'true'
... }
>>>
>>> (
... df.write
... .format("hudi")
... .options(**hudi_options)
... .save(f"{local_base_path}/{table_name}")
... )
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/work/xtable/spark-3.2.2-bin-hadoop3.2/python/pyspark/sql/readwriter.py", line 740, in save
self._jwrite.save(path)
File "/work/xtable/spark-3.2.2-bin-hadoop3.2/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
File "/work/xtable/spark-3.2.2-bin-hadoop3.2/python/pyspark/sql/utils.py", line 111, in deco
return f(*a, **kw)
File "/work/xtable/spark-3.2.2-bin-hadoop3.2/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o56.save.
: java.net.ConnectException: Call From djump/172.16.10.27 to spark-data-011:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:913)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:828)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1577)
at org.apache.hadoop.ipc.Client.call(Client.java:1519)
at org.apache.hadoop.ipc.Client.call(Client.java:1416)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:242)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:129)
at com.sun.proxy.$Proxy25.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:965)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy26.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1731)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1752)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1749)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1764)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1760)
at org.apache.hudi.HoodieSparkSqlWriter$.writeInternal(HoodieSparkSqlWriter.scala:167)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:132)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:97)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:97)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:93)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:457)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:93)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:78)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:115)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:848)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:355)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:586)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:701)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:822)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:414)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1647)
at org.apache.hadoop.ipc.Client.call(Client.java:1463)
... 65 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "daragu (via GitHub)" <gi...@apache.org>.
daragu commented on code in PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#discussion_r1600813352
##########
website/docs/how-to.md:
##########
@@ -105,7 +105,7 @@ from pyspark.sql.types import *
# initialize the bucket
table_name = "people"
-local_base_path = "/tmp/hudi-dataset"
+local_base_path = "file:/tmp/hudi-dataset"
Review Comment:
hi @the-other-tim-brown, here is a minor bug in `how-to.md` doc, so I want to make the changes together.
The intention of this example is to write a local file, but if there is no `file:/` prefix, will thrown exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "daragu (via GitHub)" <gi...@apache.org>.
daragu commented on code in PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#discussion_r1600818793
##########
website/docs/how-to.md:
##########
@@ -105,7 +105,7 @@ from pyspark.sql.types import *
# initialize the bucket
table_name = "people"
-local_base_path = "/tmp/hudi-dataset"
+local_base_path = "file:/tmp/hudi-dataset"
Review Comment:
If necessary, I would be happy to split it into two independent pull-requests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "xtable-bot (via GitHub)" <gi...@apache.org>.
xtable-bot commented on PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#issuecomment-2128854995
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=975",
"triggerID" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"triggerType" : "PUSH"
}, {
"hash" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=994",
"triggerID" : "2121345418",
"triggerType" : "MANUAL"
}, {
"hash" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=994",
"triggerID" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "2121345418",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed Azure: [SUCCESS](https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=994)
<details>
<summary>Bot commands</summary>
@xtable-bot supports the following commands:
- `@xtable-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "daragu (via GitHub)" <gi...@apache.org>.
daragu commented on PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#issuecomment-2111392143
If necessary, I would be happy to split it into two independent pull-requests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#discussion_r1600186625
##########
website/docs/how-to.md:
##########
@@ -105,7 +105,7 @@ from pyspark.sql.types import *
# initialize the bucket
table_name = "people"
-local_base_path = "/tmp/hudi-dataset"
+local_base_path = "file:/tmp/hudi-dataset"
Review Comment:
Is this meant to be part of this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "xtable-bot (via GitHub)" <gi...@apache.org>.
xtable-bot commented on PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#issuecomment-2113484846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=975",
"triggerID" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* becb98bee4e2fd133612ead830bc2ea2c98960de Azure: [CANCELED](https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=975)
<details>
<summary>Bot commands</summary>
@xtable-bot supports the following commands:
- `@xtable-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#issuecomment-2121345418
@xtable-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "xtable-bot (via GitHub)" <gi...@apache.org>.
xtable-bot commented on PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#issuecomment-2132487813
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=975",
"triggerID" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"triggerType" : "PUSH"
}, {
"hash" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=994",
"triggerID" : "2121345418",
"triggerType" : "MANUAL"
}, {
"hash" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=994",
"triggerID" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "2121345418",
"triggerType" : "MANUAL"
}, {
"hash" : "ebb4b0eec70bf4f2f6ef7d72aa5630b31fda20c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=1001",
"triggerID" : "ebb4b0eec70bf4f2f6ef7d72aa5630b31fda20c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ebb4b0eec70bf4f2f6ef7d72aa5630b31fda20c9 Azure: [SUCCESS](https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=1001)
<details>
<summary>Bot commands</summary>
@xtable-bot supports the following commands:
- `@xtable-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "xtable-bot (via GitHub)" <gi...@apache.org>.
xtable-bot commented on PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#issuecomment-2128826921
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=975",
"triggerID" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"triggerType" : "PUSH"
}, {
"hash" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2121345418",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "2121345418",
"triggerType" : "MANUAL"
}, {
"hash" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 383cf7f5aa6bbe5e4e9fea96636dcaab57bf58ed UNKNOWN
<details>
<summary>Bot commands</summary>
@xtable-bot supports the following commands:
- `@xtable-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "xtable-bot (via GitHub)" <gi...@apache.org>.
xtable-bot commented on PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#issuecomment-2113355283
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=975",
"triggerID" : "becb98bee4e2fd133612ead830bc2ea2c98960de",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* becb98bee4e2fd133612ead830bc2ea2c98960de Azure: [PENDING](https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=975)
<details>
<summary>Bot commands</summary>
@xtable-bot supports the following commands:
- `@xtable-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#discussion_r1605198751
##########
website/docs/how-to.md:
##########
@@ -105,7 +105,7 @@ from pyspark.sql.types import *
# initialize the bucket
table_name = "people"
-local_base_path = "/tmp/hudi-dataset"
+local_base_path = "file:/tmp/hudi-dataset"
Review Comment:
@daragu you run a linux OS right? @sagarlakshmipathy and I are both on mac so wondering if there is some default handling that kicks in there. @sagarlakshmipathy can you try with the `file:/` prefix and see if it still works for you. If so we can just update this so it will work for a broader audience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Demo] Make start_demo script easier to call [incubator-xtable]
Posted by "daragu (via GitHub)" <gi...@apache.org>.
daragu commented on code in PR #437:
URL: https://github.com/apache/incubator-xtable/pull/437#discussion_r1605752065
##########
website/docs/how-to.md:
##########
@@ -105,7 +105,7 @@ from pyspark.sql.types import *
# initialize the bucket
table_name = "people"
-local_base_path = "/tmp/hudi-dataset"
+local_base_path = "file:/tmp/hudi-dataset"
Review Comment:
@the-other-tim-brown, right, I run xtable in linux OS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org