You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2019/10/08 05:44:17 UTC
[jira] [Resolved] (SPARK-24784) Retraining (each document as
separate file) creates OOME
[ https://issues.apache.org/jira/browse/SPARK-24784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-24784.
----------------------------------
Resolution: Incomplete
> Retraining (each document as separate file) creates OOME
> --------------------------------------------------------
>
> Key: SPARK-24784
> URL: https://issues.apache.org/jira/browse/SPARK-24784
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.3.1
> Environment: Spark 2.3.1
> JDK8
> SBT 1.1.6
> Ubuntu 18.04
> Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
> Reporter: Jatin Puri
> Priority: Minor
> Labels: bulk-closed
>
> This looks like a bug. The reproducible can be found at: [https://github.com/purijatin/spark-retrain-bug]
> Information also available at: [http://apache-spark-developers-list.1001551.n3.nabble.com/Retraining-with-each-document-as-separate-file-creates-OOME-td24335.html]
>
> *Issue:*
> The program takes input a set of individual files (json) and each file contains a single document with the same json schema as others.
> The spark program computes the tf-idf of the documents (Tokenizer -> Stopword remover -> stemming -> tf -> tfidf).
> Once the training is complete, it unpersists and restarts the operation again.
>
>
> When run with `-Xmx1500M` memory, it fails with OOME after about 10 iterations.
>
> *Temporary Fix:*
> When all the input documents are merged to a single file, then the issue is no longer found.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org