You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Guozhang Wang <wa...@gmail.com> on 2015/05/01 20:43:18 UTC

Review Request 33761: Fix SAMZA-658

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33761/
-----------------------------------------------------------

Review request for samza.


Repository: samza


Description
-------

Add removing logic to the intermediate cached store for kv engine


Diffs
-----

  samza-kv/src/main/scala/org/apache/samza/storage/kv/BaseKeyValueStorageEngineFactory.scala b3624e6057ee1a86090f00d2853035b06f63358d 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala 61bb3f6acb080b653f8b11176538549738255acc 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/KeyValueStorageEngine.scala 3a23daf053f0b8dec3a7ec83a51c9c5527078a3b 

Diff: https://reviews.apache.org/r/33761/diff/


Testing
-------


Thanks,

Guozhang Wang


Re: Review Request 33761: Fix SAMZA-658

Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33761/#review82604
-----------------------------------------------------------


Overall looks good. Just some minor comments.


samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala
<https://reviews.apache.org/r/33761/#comment133359>

    Refer to in-memory KV-store class, we can merge those two in one function: getIter(iter) and make range() and all() referring to the same function like:
    
    def range(from: K, to: K) = {
      metrics.ranges.inc
      flush()
      getIter(store.range(from, to))
    }



samza-kv/src/test/scala/org/apache/samza/storage/kv/TestCachedStore.scala
<https://reviews.apache.org/r/33761/#comment133362>

    It would be better to test iter.remove() for both range() and all()


- Yi Pan (Data Infrastructure)


On May 4, 2015, 9:07 p.m., Guozhang Wang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33761/
> -----------------------------------------------------------
> 
> (Updated May 4, 2015, 9:07 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-658
>     https://issues.apache.org/jira/browse/SAMZA-658
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> Address Yi's comments, add unit test
> 
> 
> Diffs
> -----
> 
>   samza-kv/src/main/scala/org/apache/samza/storage/kv/BaseKeyValueStorageEngineFactory.scala b3624e6057ee1a86090f00d2853035b06f63358d 
>   samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala 61bb3f6acb080b653f8b11176538549738255acc 
>   samza-kv/src/main/scala/org/apache/samza/storage/kv/KeyValueStorageEngine.scala 3a23daf053f0b8dec3a7ec83a51c9c5527078a3b 
>   samza-kv/src/test/scala/org/apache/samza/storage/kv/MockKeyValueStore.scala PRE-CREATION 
>   samza-kv/src/test/scala/org/apache/samza/storage/kv/TestCachedStore.scala d03ec925b103ccf3c1561de0461fbc39cbe9d9ca 
> 
> Diff: https://reviews.apache.org/r/33761/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>


Re: Review Request 33761: Fix SAMZA-658

Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33761/#review82777
-----------------------------------------------------------

Ship it!


Ship It!

- Yi Pan (Data Infrastructure)


On May 6, 2015, 11:38 p.m., Guozhang Wang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33761/
> -----------------------------------------------------------
> 
> (Updated May 6, 2015, 11:38 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-658
>     https://issues.apache.org/jira/browse/SAMZA-658
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> Address Yi's comments round two
> 
> 
> Diffs
> -----
> 
>   samza-kv-inmemory/src/main/scala/org/apache/samza/storage/kv/inmemory/InMemoryKeyValueStore.scala 217333c84c696c0cc1bc3eeabf1c4066a6e89795 
>   samza-kv/src/main/scala/org/apache/samza/storage/kv/BaseKeyValueStorageEngineFactory.scala b3624e6057ee1a86090f00d2853035b06f63358d 
>   samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala 61bb3f6acb080b653f8b11176538549738255acc 
>   samza-kv/src/main/scala/org/apache/samza/storage/kv/KeyValueStorageEngine.scala 3a23daf053f0b8dec3a7ec83a51c9c5527078a3b 
>   samza-kv/src/test/scala/org/apache/samza/storage/kv/MockKeyValueStore.scala PRE-CREATION 
>   samza-kv/src/test/scala/org/apache/samza/storage/kv/TestCachedStore.scala d03ec925b103ccf3c1561de0461fbc39cbe9d9ca 
> 
> Diff: https://reviews.apache.org/r/33761/diff/
> 
> 
> Testing
> -------
> 
> unit tests
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>


Re: Review Request 33761: Fix SAMZA-658

Posted by Guozhang Wang <wa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33761/
-----------------------------------------------------------

(Updated May 7, 2015, 9:11 p.m.)


Review request for samza.


Bugs: SAMZA-658
    https://issues.apache.org/jira/browse/SAMZA-658


Repository: samza


Description (updated)
-------

Minor fix


Diffs (updated)
-----

  samza-kv-inmemory/src/main/scala/org/apache/samza/storage/kv/inmemory/InMemoryKeyValueStore.scala 23d028b3796e57a0ed16850f9a0784c02914ee1e 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/BaseKeyValueStorageEngineFactory.scala b3624e6057ee1a86090f00d2853035b06f63358d 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala 479016dff00320fa263a1a18621d1b89897adb8e 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/KeyValueStorageEngine.scala fc677b26f8ed49d3cb1ec7f46d9d494016f178af 
  samza-kv/src/test/scala/org/apache/samza/storage/kv/MockKeyValueStore.scala PRE-CREATION 
  samza-kv/src/test/scala/org/apache/samza/storage/kv/TestCachedStore.scala d03ec925b103ccf3c1561de0461fbc39cbe9d9ca 

Diff: https://reviews.apache.org/r/33761/diff/


Testing
-------

unit tests


Thanks,

Guozhang Wang


Re: Review Request 33761: Fix SAMZA-658

Posted by Guozhang Wang <wa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33761/
-----------------------------------------------------------

(Updated May 7, 2015, 5:07 p.m.)


Review request for samza.


Bugs: SAMZA-658
    https://issues.apache.org/jira/browse/SAMZA-658


Repository: samza


Description (updated)
-------

Rebase


Diffs (updated)
-----

  samza-kv-inmemory/src/main/scala/org/apache/samza/storage/kv/inmemory/InMemoryKeyValueStore.scala 23d028b3796e57a0ed16850f9a0784c02914ee1e 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/BaseKeyValueStorageEngineFactory.scala b3624e6057ee1a86090f00d2853035b06f63358d 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala 479016dff00320fa263a1a18621d1b89897adb8e 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/KeyValueStorageEngine.scala fc677b26f8ed49d3cb1ec7f46d9d494016f178af 
  samza-kv/src/test/scala/org/apache/samza/storage/kv/MockKeyValueStore.scala PRE-CREATION 
  samza-kv/src/test/scala/org/apache/samza/storage/kv/TestCachedStore.scala d03ec925b103ccf3c1561de0461fbc39cbe9d9ca 

Diff: https://reviews.apache.org/r/33761/diff/


Testing
-------

unit tests


Thanks,

Guozhang Wang


Re: Review Request 33761: Fix SAMZA-658

Posted by Guozhang Wang <wa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33761/
-----------------------------------------------------------

(Updated May 6, 2015, 11:38 p.m.)


Review request for samza.


Bugs: SAMZA-658
    https://issues.apache.org/jira/browse/SAMZA-658


Repository: samza


Description (updated)
-------

Address Yi's comments round two


Diffs (updated)
-----

  samza-kv-inmemory/src/main/scala/org/apache/samza/storage/kv/inmemory/InMemoryKeyValueStore.scala 217333c84c696c0cc1bc3eeabf1c4066a6e89795 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/BaseKeyValueStorageEngineFactory.scala b3624e6057ee1a86090f00d2853035b06f63358d 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala 61bb3f6acb080b653f8b11176538549738255acc 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/KeyValueStorageEngine.scala 3a23daf053f0b8dec3a7ec83a51c9c5527078a3b 
  samza-kv/src/test/scala/org/apache/samza/storage/kv/MockKeyValueStore.scala PRE-CREATION 
  samza-kv/src/test/scala/org/apache/samza/storage/kv/TestCachedStore.scala d03ec925b103ccf3c1561de0461fbc39cbe9d9ca 

Diff: https://reviews.apache.org/r/33761/diff/


Testing (updated)
-------

unit tests


Thanks,

Guozhang Wang


Re: Review Request 33761: Fix SAMZA-658

Posted by Guozhang Wang <wa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33761/
-----------------------------------------------------------

(Updated May 4, 2015, 9:07 p.m.)


Review request for samza.


Bugs: SAMZA-658
    https://issues.apache.org/jira/browse/SAMZA-658


Repository: samza


Description
-------

Address Yi's comments, add unit test


Diffs (updated)
-----

  samza-kv/src/main/scala/org/apache/samza/storage/kv/BaseKeyValueStorageEngineFactory.scala b3624e6057ee1a86090f00d2853035b06f63358d 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala 61bb3f6acb080b653f8b11176538549738255acc 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/KeyValueStorageEngine.scala 3a23daf053f0b8dec3a7ec83a51c9c5527078a3b 
  samza-kv/src/test/scala/org/apache/samza/storage/kv/MockKeyValueStore.scala PRE-CREATION 
  samza-kv/src/test/scala/org/apache/samza/storage/kv/TestCachedStore.scala d03ec925b103ccf3c1561de0461fbc39cbe9d9ca 

Diff: https://reviews.apache.org/r/33761/diff/


Testing
-------


Thanks,

Guozhang Wang


Re: Review Request 33761: Fix SAMZA-658

Posted by Guozhang Wang <wa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33761/
-----------------------------------------------------------

(Updated May 4, 2015, 8:56 p.m.)


Review request for samza.


Bugs: SAMZA-658
    https://issues.apache.org/jira/browse/SAMZA-658


Repository: samza


Description (updated)
-------

Address Yi's comments, add unit test


Diffs (updated)
-----

  samza-kv/src/main/scala/org/apache/samza/storage/kv/BaseKeyValueStorageEngineFactory.scala b3624e6057ee1a86090f00d2853035b06f63358d 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala 61bb3f6acb080b653f8b11176538549738255acc 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/KeyValueStorageEngine.scala 3a23daf053f0b8dec3a7ec83a51c9c5527078a3b 
  samza-kv/src/test/scala/org/apache/samza/storage/kv/TestCachedStore.scala d03ec925b103ccf3c1561de0461fbc39cbe9d9ca 

Diff: https://reviews.apache.org/r/33761/diff/


Testing
-------


Thanks,

Guozhang Wang


Re: Review Request 33761: Fix SAMZA-658

Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33761/#review82361
-----------------------------------------------------------


We would need a unit test to ensure the fix is effective. I.e. create a cached store, creating an iterator, call iterator.remove() and verify that the entry is deleted from the cache.


samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala
<https://reviews.apache.org/r/33761/#comment133093>

    This iterator needs to be wrapped as well.



samza-kv/src/main/scala/org/apache/samza/storage/kv/KeyValueStorageEngine.scala
<https://reviews.apache.org/r/33761/#comment133094>

    nit: If we are cleaning up the coding style, should be consistent w/ wrapperStore.flush() here as well.



samza-kv/src/main/scala/org/apache/samza/storage/kv/KeyValueStorageEngine.scala
<https://reviews.apache.org/r/33761/#comment133095>

    nit: same here.


- Yi Pan (Data Infrastructure)


On May 1, 2015, 6:43 p.m., Guozhang Wang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33761/
> -----------------------------------------------------------
> 
> (Updated May 1, 2015, 6:43 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-658
>     https://issues.apache.org/jira/browse/SAMZA-658
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> Add removing logic to the intermediate cached store for kv engine
> 
> 
> Diffs
> -----
> 
>   samza-kv/src/main/scala/org/apache/samza/storage/kv/BaseKeyValueStorageEngineFactory.scala b3624e6057ee1a86090f00d2853035b06f63358d 
>   samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala 61bb3f6acb080b653f8b11176538549738255acc 
>   samza-kv/src/main/scala/org/apache/samza/storage/kv/KeyValueStorageEngine.scala 3a23daf053f0b8dec3a7ec83a51c9c5527078a3b 
> 
> Diff: https://reviews.apache.org/r/33761/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>


Re: Review Request 33761: Fix SAMZA-658

Posted by Guozhang Wang <wa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33761/
-----------------------------------------------------------

(Updated May 1, 2015, 6:43 p.m.)


Review request for samza.


Bugs: SAMZA-658
    https://issues.apache.org/jira/browse/SAMZA-658


Repository: samza


Description
-------

Add removing logic to the intermediate cached store for kv engine


Diffs
-----

  samza-kv/src/main/scala/org/apache/samza/storage/kv/BaseKeyValueStorageEngineFactory.scala b3624e6057ee1a86090f00d2853035b06f63358d 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala 61bb3f6acb080b653f8b11176538549738255acc 
  samza-kv/src/main/scala/org/apache/samza/storage/kv/KeyValueStorageEngine.scala 3a23daf053f0b8dec3a7ec83a51c9c5527078a3b 

Diff: https://reviews.apache.org/r/33761/diff/


Testing
-------


Thanks,

Guozhang Wang