You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yang Jie (Jira)" <ji...@apache.org> on 2022/08/05 05:14:00 UTC
[jira] [Updated] (SPARK-39988) LevelDBIterator not close after used in `RemoteBlockPushResolver`, `YarnShuffleService` and `ExternalShuffleBlockResolver`
[ https://issues.apache.org/jira/browse/SPARK-39988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Jie updated SPARK-39988:
-----------------------------
Description:
For example:
{code:java}
@VisibleForTesting
static ConcurrentMap<AppExecId, ExecutorShuffleInfo> reloadRegisteredExecutors(DB db)
throws IOException {
ConcurrentMap<AppExecId, ExecutorShuffleInfo> registeredExecutors = Maps.newConcurrentMap();
if (db != null) {
DBIterator itr = db.iterator();
itr.seek(APP_KEY_PREFIX.getBytes(StandardCharsets.UTF_8));
while (itr.hasNext()) {
Map.Entry<byte[], byte[]> e = itr.next();
String key = new String(e.getKey(), StandardCharsets.UTF_8);
if (!key.startsWith(APP_KEY_PREFIX)) {
break;
}
AppExecId id = parseDbAppExecKey(key);
logger.info("Reloading registered executors: " + id.toString());
ExecutorShuffleInfo shuffleInfo = mapper.readValue(e.getValue(), ExecutorShuffleInfo.class);
registeredExecutors.put(id, shuffleInfo);
}
}
return registeredExecutors;
} {code}
DBIterator in above code should call close after used, otherwise, there is no other place to close it
> LevelDBIterator not close after used in `RemoteBlockPushResolver`, `YarnShuffleService` and `ExternalShuffleBlockResolver`
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-39988
> URL: https://issues.apache.org/jira/browse/SPARK-39988
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, YARN
> Affects Versions: 3.4.0
> Reporter: Yang Jie
> Priority: Major
>
> For example:
> {code:java}
> @VisibleForTesting
> static ConcurrentMap<AppExecId, ExecutorShuffleInfo> reloadRegisteredExecutors(DB db)
> throws IOException {
> ConcurrentMap<AppExecId, ExecutorShuffleInfo> registeredExecutors = Maps.newConcurrentMap();
> if (db != null) {
> DBIterator itr = db.iterator();
> itr.seek(APP_KEY_PREFIX.getBytes(StandardCharsets.UTF_8));
> while (itr.hasNext()) {
> Map.Entry<byte[], byte[]> e = itr.next();
> String key = new String(e.getKey(), StandardCharsets.UTF_8);
> if (!key.startsWith(APP_KEY_PREFIX)) {
> break;
> }
> AppExecId id = parseDbAppExecKey(key);
> logger.info("Reloading registered executors: " + id.toString());
> ExecutorShuffleInfo shuffleInfo = mapper.readValue(e.getValue(), ExecutorShuffleInfo.class);
> registeredExecutors.put(id, shuffleInfo);
> }
> }
> return registeredExecutors;
> } {code}
> DBIterator in above code should call close after used, otherwise, there is no other place to close it
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org