You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/05/30 09:07:10 UTC
[GitHub] [iceberg] flashJd opened a new issue, #4900: spark action expireSnapshots and removeOrphanFiles block in spark local mode
flashJd opened a new issue, #4900:
URL: https://github.com/apache/iceberg/issues/4900
Issue #4471 has reported this. I will add extra detail info.
precondition:
1 ./bin/spark-submit --class jobs.DeleteOrphanFilesJob --master spark://ubuntu:7077
spark standalone mode works successfully
./bin/spark-submit --class jobs.DeleteOrphanFilesJob --master local[*]
spark local mode block at last task of collectAsList operator
2. use custom hiveConfig will also cause it when run TestRemoveOrphanFilesAction3
phenomenon:
the action blocked in collectAsList and deeper stack: events.take() in spark-sql which call Condition.await()，so it is blocked here
So any suggestion why spark is blocked here and how we can solve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #4900: spark action expireSnapshots and removeOrphanFiles block in spark local mode
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #4900:
URL: https://github.com/apache/iceberg/issues/4900#issuecomment-1347558303
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] melin commented on issue #4900: spark action expireSnapshots and removeOrphanFiles block in spark local mode
Posted by GitBox <gi...@apache.org>.
melin commented on issue #4900:
URL: https://github.com/apache/iceberg/issues/4900#issuecomment-1144399901
[jstack.log](https://github.com/apache/iceberg/files/8820521/jstack.log)
@RussellSpitzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] closed issue #4900: spark action expireSnapshots and removeOrphanFiles block in spark local mode
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #4900: spark action expireSnapshots and removeOrphanFiles block in spark local mode
URL: https://github.com/apache/iceberg/issues/4900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #4900: spark action expireSnapshots and removeOrphanFiles block in spark local mode
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #4900:
URL: https://github.com/apache/iceberg/issues/4900#issuecomment-1143071616
Did you check a thread dump? What makes you think it is stuck?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #4900: spark action expireSnapshots and removeOrphanFiles block in spark local mode
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #4900:
URL: https://github.com/apache/iceberg/issues/4900#issuecomment-1372958547
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #4900: spark action expireSnapshots and removeOrphanFiles block in spark local mode
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #4900:
URL: https://github.com/apache/iceberg/issues/4900#issuecomment-1144428272
Inside the broadcast exchange thread also seems to be waiting, I'd check out the stage view, is that a single final task we are waiting for? Or another stage. In the dump I didn't see any executor threads running anything but I didn't look that close.
I would also just try disabling
1. Adaptive query planning
2. Broadcast joins
Could be that to broadcast is just not completing due to memory pressure, but nothing in that dump looks iceberg specific to me. Just lots of Spark internals waiting on other spark internals
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on issue #4900: spark action expireSnapshots and removeOrphanFiles block in spark local mode
Posted by GitBox <gi...@apache.org>.
amogh-jahagirdar commented on issue #4900:
URL: https://github.com/apache/iceberg/issues/4900#issuecomment-1142703277
I think is expected as @RussellSpitzer mentioned in https://github.com/apache/iceberg/issues/4471#issuecomment-1086660475. The spark procedure has to wait until the computation of the files to actually be deleted. Currently, the results of that computation is collected to the spark driver, and deletion of the files occurs on the driver. Even if the procedure did a distributed deletion of files (not currently done due to concerns of more easily hitting rate limiting from the underlying FS), the procedure needs to first compute the files to be deleted. So if that's the bottleneck in the collectAsList, then that's expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flashJd commented on issue #4900: spark action expireSnapshots and removeOrphanFiles block in spark local mode
Posted by GitBox <gi...@apache.org>.
flashJd commented on issue #4900:
URL: https://github.com/apache/iceberg/issues/4900#issuecomment-1143022128
It seems not like a performance bottleneck but a completely stuck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] eric666666 commented on issue #4900: spark action expireSnapshots and removeOrphanFiles block in spark local mode
Posted by GitBox <gi...@apache.org>.
eric666666 commented on issue #4900:
URL: https://github.com/apache/iceberg/issues/4900#issuecomment-1155912241
I fixed this problem. You can see issue #4471
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org