You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/02/03 17:34:15 UTC
[GitHub] [iceberg] szehon-ho commented on issue #4035: Writing a Spark Repartition-ed Dataframe into Iceberg Table goes into infinite loop
szehon-ho commented on issue #4035:
URL: https://github.com/apache/iceberg/issues/4035#issuecomment-1029231888
Looks like in the log like it's actually making steady progress (up to 123/200 executors when it finished)
If you are running locally maybe there are just too many partitions after the repartition and not enough resources?
```
22/02/03 19:04:07 INFO Executor: Running task 117.0 in stage 1.0 (TID 121)
22/02/03 19:04:07 INFO TaskSetManager: Finished task 116.0 in stage 1.0 (TID 120) in 1385 ms on LT0089527.in.ril.com (executor driver) (120/200)
22/02/03 19:04:07 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
22/02/03 19:04:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/03 19:04:07 INFO CodecPool: Got brand-new compressor [.gz]
22/02/03 19:04:09 INFO DataWritingSparkTask: Commit authorized for partition 117 (task 121, attempt 0, stage 1.0)
22/02/03 19:04:09 INFO DataWritingSparkTask: Committed partition 117 (task 121, attempt 0, stage 1.0)
22/02/03 19:04:09 INFO Executor: Finished task 117.0 in stage 1.0 (TID 121). 2462 bytes result sent to driver
22/02/03 19:04:09 INFO TaskSetManager: Starting task 118.0 in stage 1.0 (TID 122, LT0089527.in.ril.com, executor driver, partition 118, PROCESS_LOCAL, 7325 bytes)
22/02/03 19:04:09 INFO TaskSetManager: Finished task 117.0 in stage 1.0 (TID 121) in 1321 ms on LT0089527.in.ril.com (executor driver) (121/200)
22/02/03 19:04:09 INFO Executor: Running task 118.0 in stage 1.0 (TID 122)
22/02/03 19:04:09 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
22/02/03 19:04:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
22/02/03 19:04:09 INFO CodecPool: Got brand-new compressor [.gz]
22/02/03 19:04:10 INFO DataWritingSparkTask: Commit authorized for partition 118 (task 122, attempt 0, stage 1.0)
22/02/03 19:04:10 INFO DataWritingSparkTask: Committed partition 118 (task 122, attempt 0, stage 1.0)
22/02/03 19:04:10 INFO Executor: Finished task 118.0 in stage 1.0 (TID 122). 2462 bytes result sent to driver
22/02/03 19:04:10 INFO TaskSetManager: Starting task 119.0 in stage 1.0 (TID 123, LT0089527.in.ril.com, executor driver, partition 119, PROCESS_LOCAL, 7325 bytes)
22/02/03 19:04:10 INFO Executor: Running task 119.0 in stage 1.0 (TID 123)
22/02/03 19:04:10 INFO TaskSetManager: Finished task 118.0 in stage 1.0 (TID 122) in 1409 ms on LT0089527.in.ril.com (executor driver) (122/200)
22/02/03 19:04:10 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
22/02/03 19:04:10 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/03 19:04:10 INFO CodecPool: Got brand-new compressor [.gz]
22/02/03 19:04:12 INFO DataWritingSparkTask: Commit authorized for partition 119 (task 123, attempt 0, stage 1.0)
22/02/03 19:04:12 INFO DataWritingSparkTask: Committed partition 119 (task 123, attempt 0, stage 1.0)
22/02/03 19:04:12 INFO Executor: Finished task 119.0 in stage 1.0 (TID 123). 2462 bytes result sent to driver
22/02/03 19:04:12 INFO TaskSetManager: Starting task 120.0 in stage 1.0 (TID 124, LT0089527.in.ril.com, executor driver, partition 120, PROCESS_LOCAL, 7325 bytes)
22/02/03 19:04:12 INFO Executor: Running task 120.0 in stage 1.0 (TID 124)
22/02/03 19:04:12 INFO TaskSetManager: Finished task 119.0 in stage 1.0 (TID 123) in 2072 ms on LT0089527.in.ril.com (executor driver) (123/200)
22/02/03 19:04:12 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
22/02/03 19:04:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/03 19:04:12 INFO CodecPool: Got brand-new compressor [.gz]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org