[jira] [Created] (SPARK-20448) Document how FileInputDStream works with object storage

["Steve Loughran (JIRA)" <ji...@apache.org>]

Steve Loughran created SPARK-20448:
--------------------------------------

             Summary: Document how FileInputDStream works with object storage
                 Key: SPARK-20448
                 URL: https://issues.apache.org/jira/browse/SPARK-20448
             Project: Spark
          Issue Type: Documentation
          Components: Documentation
    Affects Versions: 2.1.0
            Reporter: Steve Loughran
            Priority: Minor


Object stores work differently from filesystems: intermediate writes not visible, renames are really O(data) copies, not O(1) transactions.

This makes working with them as DStreams fundamentally different: you can write straight into the destination.

1. Document how FileinputDStreams scan directories for changes
2. Document how object stores behave differently, and the implications
for users.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org