You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/10/25 09:12:03 UTC
[GitHub] [iceberg] chirenwang opened a new issue, #6050: Spark3.2.2 rewriteDataFiles task yarn driver Stuck
chirenwang opened a new issue, #6050:
URL: https://github.com/apache/iceberg/issues/6050
### Apache Iceberg version
0.14.0
### Query engine
Spark
### Please describe the bug 🐞
I use SparkActions.rewriteDataFiles to compact my small files, this is my code
String namespace = "dwm_stk";
SparkSession sparkSession = SparkSession.builder()
.config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
.config("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.catalog.spark_catalog.type", "hive")
.config("spark.sql.catalog.local", "org.apache.iceberg.spark.SparkCatalog")
.config("spark.sql.catalog.local.type", "hadoop")
.config("spark.sql.catalog.local.warehouse", "$PWD/warehouse")
.config("iceberg.engine.hive.enabled", "true")
.config("spark.sql.autoBroadcastJoinThreshold", "-1")
.config("spark.shuffle.service.enabled", "true")
.config("spark.dynamicAllocation.enabled", "true")
.config("spark.dynamicAllocation.initialExecutors", "1")
.config("spark.dynamicAllocation.maxExecutors", "10")
.config("spark.shuffle.service.port", "7337")
.config("spark.sql.adaptive.enabled", "true")
.config("spark.python.authenticate.socketTimeout", "120s")
.enableHiveSupport()
.getOrCreate();
HiveCatalog catalog = new HiveCatalog();
catalog.setConf(sparkSession.sparkContext().hadoopConfiguration());
Map<String, String> properties = new HashMap<>();
catalog.initialize("hive", properties);
SparkActions sparkActions = SparkActions.get(sparkSession);
String[] tables = new String[]{
"mt_bigbuyorder", "mt_bigsellorder",
"mt_bigtotalorder",
"mt_bigwillingbuyorder", "mt_bigwillingsellorder",
"mt_bigwillingtotalorder", "mt_buyorder", "mt_min_bigbuyorder",
"mt_min_bigsellorder", "mt_min_bigtotalorder", "mt_min_bigwillingbuyorder",
"mt_min_bigwillingsellorder", "mt_min_bigwillingtotalorder", "mt_min_buyorder",
"mt_min_sellorder", "mt_min_totalorder", "mt_min_willingbuyorder",
"mt_min_willingsellorder", "mt_min_willingtotalorder", "mt_sellorder",
"mt_totalorder", "mt_willingbuyorder", "mt_willingsellorder", "mt_willingtotalorder"
};
for (String tableString : tables) {
Table table = catalog.loadTable(TableIdentifier.of(namespace, tableString));
long tsToDelete = System.currentTimeMillis();
sparkActions.deleteOrphanFiles(table).olderThan(tsToDelete).execute();
sparkActions.rewriteDataFiles(table)
.option("target-file-size-bytes", Long.toString(128 * 1024 * 1024)) // 128 MB
.option("max-concurrent-file-group-rewrites", "20")
.option("delete-file-threshold", "1")
.execute();
long tsToExpire = System.currentTimeMillis();
sparkActions
.expireSnapshots(table)
.expireOlderThan(tsToExpire)
.execute();
}
my question is my rewriteDataFiles task would stuck when it have finished 1.5k job, with all the spark worker down and driver process cost high cpu over to 98% cpu of my server even I can't use jstack <pid> to get the driver's threaddump.
I doubt that may be an endless loop bug in SparkActions.rewriteDataFiles.
please help me fix it, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] deadwind4 commented on issue #6050: Spark3.2.2 rewriteDataFiles task yarn driver Stuck
Posted by GitBox <gi...@apache.org>.
deadwind4 commented on issue #6050:
URL: https://github.com/apache/iceberg/issues/6050#issuecomment-1299451525
https://github.com/apache/iceberg/issues/4471
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6050: Spark3.2.2 rewriteDataFiles task yarn driver Stuck
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6050:
URL: https://github.com/apache/iceberg/issues/6050#issuecomment-1530637499
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] closed issue #6050: Spark3.2.2 rewriteDataFiles task yarn driver Stuck
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6050: Spark3.2.2 rewriteDataFiles task yarn driver Stuck
URL: https://github.com/apache/iceberg/issues/6050
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] chirenwang commented on issue #6050: Spark3.2.2 rewriteDataFiles task yarn driver Stuck
Posted by GitBox <gi...@apache.org>.
chirenwang commented on issue #6050:
URL: https://github.com/apache/iceberg/issues/6050#issuecomment-1290287159
I fix this problem by imporve my spark driver cores and memory ,from 1core 4G to 4cores 16G, by I don't know why
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6050: Spark3.2.2 rewriteDataFiles task yarn driver Stuck
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6050:
URL: https://github.com/apache/iceberg/issues/6050#issuecomment-1556047979
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org