[jira] [Created] (SPARK-8024) Luigi triggering resolved Blockmanager bug

["Cory Nguyen (JIRA)" <ji...@apache.org>]

Cory Nguyen created SPARK-8024:
----------------------------------

             Summary: Luigi triggering resolved Blockmanager bug
                 Key: SPARK-8024
                 URL: https://issues.apache.org/jira/browse/SPARK-8024
             Project: Spark
          Issue Type: Bug
          Components: Block Manager
    Affects Versions: 1.3.1
            Reporter: Cory Nguyen


We are using Luigi with Spark to manage our jobs

However we run into a unique rare case with the following conditions that trigger the resolved Block Manger Bug:

- Dataset is relatively large ~ 1.5TB
- Spark job is ran with Luigi
- save to local HDFS

The spark job would process data and mappings just fine, until the very end when it proceeds to save the files to local hdfs this is when it triggers this bug. 

However, the job saves and complete data successfully if it was saved to s3:// location.

wondering what might cause this resolved bug to trigger when ran with luigi saving to local hdfs but not trigger when saved to s3 with luigi or ran without luigi?





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org