[jira] [Commented] (BAHIR-110) Implement _changes API for non-streaming receiver

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BAHIR-110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16097919#comment-16097919 ] 

ASF GitHub Bot commented on BAHIR-110:
--------------------------------------

Github user emlaver commented on the issue:

    https://github.com/apache/bahir/pull/45
  
    @mayya-sharipova Thank you for all the time you've spent reviewing this work.
    The storageLevel option was not working as expected and also was not in the correct section in the README.  I’ve renamed the option to `cloudant.storageLevel` (as this option only works when _changes endpoint option is set), updated the README, and successfully tested and verified that the option is working using spark-submit and SparkSession’s config in 052abb4.  I've run the Scala tests and examples and they all passed.  Could you please review and approve these changes?


> Implement _changes API for non-streaming receiver
> -------------------------------------------------
>
>                 Key: BAHIR-110
>                 URL: https://issues.apache.org/jira/browse/BAHIR-110
>             Project: Bahir
>          Issue Type: Improvement
>            Reporter: Esteban Laver
>   Original Estimate: 216h
>  Remaining Estimate: 216h
>
> Today we use the _changes API for Spark streaming receiver and _all_docs API for non-streaming receiver. _all_docs API supports parallel reads (using offset and range) but performance of _changes API is still better in most cases (even with single threaded support).
> With this ticket we want to:
> a) implement _changes API for non-streaming receivers
> b) allow customers to pick either _all_docs (default) or _changes API endpoint, with documentation about pros and cons
> _changes performance details:
> Successfully loaded Cloudant (using local cloudant-developer docker image) docs into Spark (local standalone) with the following database sizes: 15GB (time: 8 1/2 mins), 20GB (17 mins), 46GB (25 mins), and 75GB (48 1/2 mins).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)