You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Navina Ramesh <nr...@linkedin.com> on 2015/04/22 23:47:49 UTC
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/
-----------------------------------------------------------
(Updated April 22, 2015, 9:47 p.m.)
Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
Summary (updated)
-----------------
SAMZA-557 Reuse local state in SamzaContainer on clean shutdown
Repository: samza
Description
-------
Changed default to yarn cwd instead of io.tmpDir and refactored code
Diffs
-----
samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 720fbdceafc4fe69b048d81a677e874d13e6d22f
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 66c2a0dc2e38e21f951727a30f0987776ac52fe2
Diff: https://reviews.apache.org/r/33453/diff/
Testing (updated)
-------
Tested locally using hello-samza.
Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
Thanks,
Navina Ramesh
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Navina Ramesh <nr...@linkedin.com>.
> On April 28, 2015, 5:43 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala, line 457
> > <https://reviews.apache.org/r/33453/diff/2/?file=939981#file939981line457>
> >
> > Question: shouldn't the base dir to be determined by container ID, not jobId?
>
> Navina Ramesh wrote:
> I think containerId is not really relevant because the store partition directories are uniquely named across the containers. The ownership of containers and partition directory is persisted in the config stream (going forward). I will verify this and confirm here.
Verified. Store partitions have unique names and are agnostic of the container to which they belong.
- Navina
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/#review81838
-----------------------------------------------------------
On April 22, 2015, 9:54 p.m., Navina Ramesh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33453/
> -----------------------------------------------------------
>
> (Updated April 22, 2015, 9:54 p.m.)
>
>
> Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Changed default to yarn cwd instead of io.tmpDir and refactored code
>
>
> Diffs
> -----
>
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 720fbdceafc4fe69b048d81a677e874d13e6d22f
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 66c2a0dc2e38e21f951727a30f0987776ac52fe2
>
> Diff: https://reviews.apache.org/r/33453/diff/
>
>
> Testing
> -------
>
> Tested locally using hello-samza.
> Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
>
>
> Thanks,
>
> Navina Ramesh
>
>
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Navina Ramesh <nr...@linkedin.com>.
> On April 28, 2015, 5:43 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala, line 88
> > <https://reviews.apache.org/r/33453/diff/2/?file=939982#file939982line88>
> >
> > qq: Is there any checksum in the offset file to make sure that we don't read in a corrupted value?
>
> Navina Ramesh wrote:
> No. As of now, we don't have any checksum. Do you think it is better to add it now than later ?
Yi,
I created SAMZA-666 to work on adding checksum data to the offset file.
- Navina
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/#review81838
-----------------------------------------------------------
On May 4, 2015, 8:57 p.m., Navina Ramesh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33453/
> -----------------------------------------------------------
>
> (Updated May 4, 2015, 8:57 p.m.)
>
>
> Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Added Unit Tests for TaskStorageManager and refactored some code
>
> Changed default to yarn cwd instead of io.tmpDir and refactored code
>
>
> Diffs
> -----
>
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala e94a4735217f59d074510ce1556c8c439e6a72f0
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ac4793afe1e6868933e750181bee1e27c157b5e6
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
> samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala PRE-CREATION
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 1b44a517129b35affac802929087eaa0061e6b5d
>
> Diff: https://reviews.apache.org/r/33453/diff/
>
>
> Testing
> -------
>
> Tested locally using hello-samza.
> Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
>
>
> Thanks,
>
> Navina Ramesh
>
>
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Navina Ramesh <nr...@linkedin.com>.
> On April 28, 2015, 5:43 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala, line 457
> > <https://reviews.apache.org/r/33453/diff/2/?file=939981#file939981line457>
> >
> > Question: shouldn't the base dir to be determined by container ID, not jobId?
I think containerId is not really relevant because the store partition directories are uniquely named across the containers. The ownership of containers and partition directory is persisted in the config stream (going forward). I will verify this and confirm here.
> On April 28, 2015, 5:43 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala, line 483
> > <https://reviews.apache.org/r/33453/diff/2/?file=939981#file939981line483>
> >
> > nit: I think that the readablity is a little better if:
> >
> > val storeBaseDir = if (changeLogSystemStreamPartition != null) {
> > loggedStorageBaseDir
> > } else {
> > defaultStoreBaseDir
> > }
> > ...
> > val storageEngine = ... (
> > storeName,
> > TaskStorageManager.getStorePartitionDir(storeBaseDir, storeName, taskName),
> > ...
Ok. Agreed.
> On April 28, 2015, 5:43 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala, line 76
> > <https://reviews.apache.org/r/33453/diff/2/?file=939982#file939982line76>
> >
> > How do we know that this store is using the default storeBaseDir not loggedStoreBaseDir?
At this point, we don't check if the store is using default storeBaseDir or loggedStoreBaseDir. Since the application could have used default storeBaseDir in the past, we are being a bit pessimistic and cleaning up any residual storage directories.
I am not very happy with the clean up code. It is generic enough to be agnostic of the physical deployment. But this means it is doing additional work of deleting the directory which won't be needed otherwise. For example, if you are deploying using YARN, each application run is contained in its own app directory. Here, the "cleanBaseDir" step effectively only creates the storage directory (Line 72 to 77).
> On April 28, 2015, 5:43 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala, line 88
> > <https://reviews.apache.org/r/33453/diff/2/?file=939982#file939982line88>
> >
> > qq: Is there any checksum in the offset file to make sure that we don't read in a corrupted value?
No. As of now, we don't have any checksum. Do you think it is better to add it now than later ?
> On April 28, 2015, 5:43 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala, line 99
> > <https://reviews.apache.org/r/33453/diff/2/?file=939982#file939982line99>
> >
> > From the store initiation code, it seems that storeBaseDir won't have any logged store paths if logged store base dir is configured. Hence, here we may be deleting empty/non-existing storagePartitionDirs.
True. That is exactly what I meant in comment above. We can probably modify the Util.rm method to be a no-op when the file path doesn't exist.
- Navina
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/#review81838
-----------------------------------------------------------
On April 22, 2015, 9:54 p.m., Navina Ramesh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33453/
> -----------------------------------------------------------
>
> (Updated April 22, 2015, 9:54 p.m.)
>
>
> Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Changed default to yarn cwd instead of io.tmpDir and refactored code
>
>
> Diffs
> -----
>
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 720fbdceafc4fe69b048d81a677e874d13e6d22f
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 66c2a0dc2e38e21f951727a30f0987776ac52fe2
>
> Diff: https://reviews.apache.org/r/33453/diff/
>
>
> Testing
> -------
>
> Tested locally using hello-samza.
> Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
>
>
> Thanks,
>
> Navina Ramesh
>
>
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/#review81838
-----------------------------------------------------------
Overall looks good to me. It would be nice to add unit tests as well. Thanks!
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala
<https://reviews.apache.org/r/33453/#comment132322>
Question: shouldn't the base dir to be determined by container ID, not jobId?
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala
<https://reviews.apache.org/r/33453/#comment132326>
nit: I think that the readablity is a little better if:
val storeBaseDir = if (changeLogSystemStreamPartition != null) {
loggedStorageBaseDir
} else {
defaultStoreBaseDir
}
...
val storageEngine = ... (
storeName,
TaskStorageManager.getStorePartitionDir(storeBaseDir, storeName, taskName),
...
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala
<https://reviews.apache.org/r/33453/#comment132329>
How do we know that this store is using the default storeBaseDir not loggedStoreBaseDir?
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala
<https://reviews.apache.org/r/33453/#comment132331>
qq: Is there any checksum in the offset file to make sure that we don't read in a corrupted value?
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala
<https://reviews.apache.org/r/33453/#comment132330>
From the store initiation code, it seems that storeBaseDir won't have any logged store paths if logged store base dir is configured. Hence, here we may be deleting empty/non-existing storagePartitionDirs.
- Yi Pan (Data Infrastructure)
On April 22, 2015, 9:54 p.m., Navina Ramesh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33453/
> -----------------------------------------------------------
>
> (Updated April 22, 2015, 9:54 p.m.)
>
>
> Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Changed default to yarn cwd instead of io.tmpDir and refactored code
>
>
> Diffs
> -----
>
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 720fbdceafc4fe69b048d81a677e874d13e6d22f
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 66c2a0dc2e38e21f951727a30f0987776ac52fe2
>
> Diff: https://reviews.apache.org/r/33453/diff/
>
>
> Testing
> -------
>
> Tested locally using hello-samza.
> Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
>
>
> Thanks,
>
> Navina Ramesh
>
>
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Navina Ramesh <nr...@linkedin.com>.
> On May 5, 2015, 11:21 p.m., Naveen Somasundaram wrote:
> > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala, line 451
> > <https://reviews.apache.org/r/33453/diff/4/?file=950515#file950515line451>
> >
> > nit: Prefer changelogged over logged, as it seems consistent with other usages in our code.
I don't see any "changelogged" usage. It actually makes the variable name really long.
Since loggedStore can't really mean anything other than a store with changelog, I think it should remain as is.
- Navina
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/#review82593
-----------------------------------------------------------
On May 5, 2015, 6:47 p.m., Navina Ramesh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33453/
> -----------------------------------------------------------
>
> (Updated May 5, 2015, 6:47 p.m.)
>
>
> Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Added checksum to the Offset file and some unit tests
>
> Added Unit Tests for TaskStorageManager and refactored some code
>
> Changed default to yarn cwd instead of io.tmpDir and refactored code
>
>
> Diffs
> -----
>
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala e94a4735217f59d074510ce1556c8c439e6a72f0
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ac4793afe1e6868933e750181bee1e27c157b5e6
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 8a83566ae6139127d7fe04ab42231151227dc479
> samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala PRE-CREATION
> samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala b75f44060fb8e660e824eaeb9cfdcc9d6fa902e8
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 1b44a517129b35affac802929087eaa0061e6b5d
>
> Diff: https://reviews.apache.org/r/33453/diff/
>
>
> Testing
> -------
>
> Tested locally using hello-samza.
> Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
>
>
> Thanks,
>
> Navina Ramesh
>
>
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Navina Ramesh <nr...@linkedin.com>.
> On May 5, 2015, 11:21 p.m., Naveen Somasundaram wrote:
> > samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala, line 55
> > <https://reviews.apache.org/r/33453/diff/4/?file=950514#file950514line55>
> >
> > Can you also please update the configuration docs (or someother docs where it would make sense) ?
I have created [SAMZA-668|https://issues.apache.org/jira/browse/SAMZA-668] for adding documentation.
> On May 5, 2015, 11:21 p.m., Naveen Somasundaram wrote:
> > samza-core/src/main/scala/org/apache/samza/util/Util.scala, line 274
> > <https://reviews.apache.org/r/33453/diff/4/?file=950517#file950517line274>
> >
> > nit: log ("CRC doesn't match, data corrupted, skipping file") or something along those lines.
Ok. Makes sense.
- Navina
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/#review82593
-----------------------------------------------------------
On May 5, 2015, 6:47 p.m., Navina Ramesh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33453/
> -----------------------------------------------------------
>
> (Updated May 5, 2015, 6:47 p.m.)
>
>
> Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Added checksum to the Offset file and some unit tests
>
> Added Unit Tests for TaskStorageManager and refactored some code
>
> Changed default to yarn cwd instead of io.tmpDir and refactored code
>
>
> Diffs
> -----
>
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala e94a4735217f59d074510ce1556c8c439e6a72f0
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ac4793afe1e6868933e750181bee1e27c157b5e6
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 8a83566ae6139127d7fe04ab42231151227dc479
> samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala PRE-CREATION
> samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala b75f44060fb8e660e824eaeb9cfdcc9d6fa902e8
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 1b44a517129b35affac802929087eaa0061e6b5d
>
> Diff: https://reviews.apache.org/r/33453/diff/
>
>
> Testing
> -------
>
> Tested locally using hello-samza.
> Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
>
>
> Thanks,
>
> Navina Ramesh
>
>
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Naveen Somasundaram <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/#review82593
-----------------------------------------------------------
samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala
<https://reviews.apache.org/r/33453/#comment133344>
Can you also please update the configuration docs (or someother docs where it would make sense) ?
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala
<https://reviews.apache.org/r/33453/#comment133345>
nit: Prefer changelogged over logged, as it seems consistent with other usages in our code.
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala
<https://reviews.apache.org/r/33453/#comment133353>
Idiomatic scala: (taskStores.keySet -- loggedStores)?
samza-core/src/main/scala/org/apache/samza/util/Util.scala
<https://reviews.apache.org/r/33453/#comment133363>
nit: log ("CRC doesn't match, data corrupted, skipping file") or something along those lines.
- Naveen Somasundaram
On May 5, 2015, 6:47 p.m., Navina Ramesh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33453/
> -----------------------------------------------------------
>
> (Updated May 5, 2015, 6:47 p.m.)
>
>
> Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Added checksum to the Offset file and some unit tests
>
> Added Unit Tests for TaskStorageManager and refactored some code
>
> Changed default to yarn cwd instead of io.tmpDir and refactored code
>
>
> Diffs
> -----
>
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala e94a4735217f59d074510ce1556c8c439e6a72f0
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ac4793afe1e6868933e750181bee1e27c157b5e6
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 8a83566ae6139127d7fe04ab42231151227dc479
> samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala PRE-CREATION
> samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala b75f44060fb8e660e824eaeb9cfdcc9d6fa902e8
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 1b44a517129b35affac802929087eaa0061e6b5d
>
> Diff: https://reviews.apache.org/r/33453/diff/
>
>
> Testing
> -------
>
> Tested locally using hello-samza.
> Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
>
>
> Thanks,
>
> Navina Ramesh
>
>
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Navina Ramesh <nr...@linkedin.com>.
> On May 6, 2015, 12:10 a.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala, lines 481-486
> > <https://reviews.apache.org/r/33453/diff/4/?file=950515#file950515line481>
> >
> > 1. why does the changeLogSystemStreamPartition matter here?
> > 2. prefer the storePartitionDir, because it's the dir for each partition/task, right?
>
> Navina Ramesh wrote:
> 1. why does the changeLogSystemStreamPartition matter here?
> >> changeLogSystemStreamPartition indicates whether a store has a changelog or not. Depending on the case, the base directory of the store will be different.
>
> 2. prefer the storePartitionDir, because it's the dir for each partition/task, right?
> >> Base directory is same for all tasks in the job. There is a separate partition directory for each task within each store directory. The path looks like : "<StoreBaseDir>/<StoreName>/<TaskName>/<storeFiles>" . Does that clarify your question?
>
> Yan Fang wrote:
> 1. "changeLogSystemStreamPartition indicates whether a store has a changelog or not. Depending on the case, the base directory of the store will be different."
> It sounds a little confusing to me. Don't all stores have changelog? Do you mean offset here?
>
> 2. for "TaskStorageManager.getStorePartitionDir(loggedStorageBaseDir, storeName, taskName)", if two jobs have the same storeName and taskName, will they be in the same directory?
It sounds a little confusing to me. Don't all stores have changelog? Do you mean offset here?
> Not all stores have changelog. You have to configure a store to have a changelog using "stores.<storename>.changelog" property. It will be expensive to associate a changelog with all stores when a job doesn't care about losing its state.
for "TaskStorageManager.getStorePartitionDir(loggedStorageBaseDir, storeName, taskName)", if two jobs have the same storeName and taskName, will they be in the same directory?
> No. The storeBaseDir is specific to the job. To clarify, the path above looks more like: "<workingDir>/$jobName-$jobId/$storeName/$taskName/<storeFiles> .
- Navina
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/#review82610
-----------------------------------------------------------
On May 6, 2015, 6:22 a.m., Navina Ramesh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33453/
> -----------------------------------------------------------
>
> (Updated May 6, 2015, 6:22 a.m.)
>
>
> Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Added checksum to the Offset file and some unit tests
>
> Added Unit Tests for TaskStorageManager and refactored some code
>
> Changed default to yarn cwd instead of io.tmpDir and refactored code
>
>
> Diffs
> -----
>
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala e94a4735217f59d074510ce1556c8c439e6a72f0
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ac4793afe1e6868933e750181bee1e27c157b5e6
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 8a83566ae6139127d7fe04ab42231151227dc479
> samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala PRE-CREATION
> samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala b75f44060fb8e660e824eaeb9cfdcc9d6fa902e8
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 1b44a517129b35affac802929087eaa0061e6b5d
>
> Diff: https://reviews.apache.org/r/33453/diff/
>
>
> Testing
> -------
>
> Tested locally using hello-samza.
> Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
>
>
> Thanks,
>
> Navina Ramesh
>
>
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Yan Fang <ya...@gmail.com>.
> On May 6, 2015, 12:10 a.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala, lines 481-486
> > <https://reviews.apache.org/r/33453/diff/4/?file=950515#file950515line481>
> >
> > 1. why does the changeLogSystemStreamPartition matter here?
> > 2. prefer the storePartitionDir, because it's the dir for each partition/task, right?
>
> Navina Ramesh wrote:
> 1. why does the changeLogSystemStreamPartition matter here?
> >> changeLogSystemStreamPartition indicates whether a store has a changelog or not. Depending on the case, the base directory of the store will be different.
>
> 2. prefer the storePartitionDir, because it's the dir for each partition/task, right?
> >> Base directory is same for all tasks in the job. There is a separate partition directory for each task within each store directory. The path looks like : "<StoreBaseDir>/<StoreName>/<TaskName>/<storeFiles>" . Does that clarify your question?
1. "changeLogSystemStreamPartition indicates whether a store has a changelog or not. Depending on the case, the base directory of the store will be different."
It sounds a little confusing to me. Don't all stores have changelog? Do you mean offset here?
2. for "TaskStorageManager.getStorePartitionDir(loggedStorageBaseDir, storeName, taskName)", if two jobs have the same storeName and taskName, will they be in the same directory?
> On May 6, 2015, 12:10 a.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala, lines 110-112
> > <https://reviews.apache.org/r/33453/diff/4/?file=950516#file950516line110>
> >
> > personal opinion: should this logic be another method, such as readOffsetFromFile, not "cleanBaseDir"? A little misleading.
>
> Navina Ramesh wrote:
> I can modularize it further if it helps. Since we need to load offsets (if available) before cleaning directories, it became a part of it.
> Let me fix this.
>
> Navina Ramesh wrote:
> Actually, since the decision to delete the directory depends on the offset value, I don't see any gain in moving that call to a separate method. It might make the code more complicated by adding extraneous variable.
I see. understood.
- Yan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/#review82610
-----------------------------------------------------------
On May 6, 2015, 6:22 a.m., Navina Ramesh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33453/
> -----------------------------------------------------------
>
> (Updated May 6, 2015, 6:22 a.m.)
>
>
> Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Added checksum to the Offset file and some unit tests
>
> Added Unit Tests for TaskStorageManager and refactored some code
>
> Changed default to yarn cwd instead of io.tmpDir and refactored code
>
>
> Diffs
> -----
>
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala e94a4735217f59d074510ce1556c8c439e6a72f0
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ac4793afe1e6868933e750181bee1e27c157b5e6
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 8a83566ae6139127d7fe04ab42231151227dc479
> samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala PRE-CREATION
> samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala b75f44060fb8e660e824eaeb9cfdcc9d6fa902e8
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 1b44a517129b35affac802929087eaa0061e6b5d
>
> Diff: https://reviews.apache.org/r/33453/diff/
>
>
> Testing
> -------
>
> Tested locally using hello-samza.
> Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
>
>
> Thanks,
>
> Navina Ramesh
>
>
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Navina Ramesh <nr...@linkedin.com>.
> On May 6, 2015, 12:10 a.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala, lines 110-112
> > <https://reviews.apache.org/r/33453/diff/4/?file=950516#file950516line110>
> >
> > personal opinion: should this logic be another method, such as readOffsetFromFile, not "cleanBaseDir"? A little misleading.
>
> Navina Ramesh wrote:
> I can modularize it further if it helps. Since we need to load offsets (if available) before cleaning directories, it became a part of it.
> Let me fix this.
Actually, since the decision to delete the directory depends on the offset value, I don't see any gain in moving that call to a separate method. It might make the code more complicated by adding extraneous variable.
- Navina
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/#review82610
-----------------------------------------------------------
On May 5, 2015, 6:47 p.m., Navina Ramesh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33453/
> -----------------------------------------------------------
>
> (Updated May 5, 2015, 6:47 p.m.)
>
>
> Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Added checksum to the Offset file and some unit tests
>
> Added Unit Tests for TaskStorageManager and refactored some code
>
> Changed default to yarn cwd instead of io.tmpDir and refactored code
>
>
> Diffs
> -----
>
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala e94a4735217f59d074510ce1556c8c439e6a72f0
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ac4793afe1e6868933e750181bee1e27c157b5e6
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 8a83566ae6139127d7fe04ab42231151227dc479
> samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala PRE-CREATION
> samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala b75f44060fb8e660e824eaeb9cfdcc9d6fa902e8
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 1b44a517129b35affac802929087eaa0061e6b5d
>
> Diff: https://reviews.apache.org/r/33453/diff/
>
>
> Testing
> -------
>
> Tested locally using hello-samza.
> Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
>
>
> Thanks,
>
> Navina Ramesh
>
>
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Navina Ramesh <nr...@linkedin.com>.
> On May 6, 2015, 12:10 a.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala, line 55
> > <https://reviews.apache.org/r/33453/diff/4/?file=950514#file950514line55>
> >
> > 1.is it also possible to retrived this value from the config? Then users do not need to set it in every node.
> > 2.out of curious, why is it "logged" and "non-logged", not something like, "persisting" and "non-persisting"?
1.is it also possible to retrived this value from the config? Then users do not need to set it in every node.
>> Essentially, we don't want this to be configured at a "per-job" level. If we do so, then we cannot clean-up after the stale data stores (ie. data stores that are not longer being used by any task). From my understanding, the only way to set a configuration at the cluster level is through environment variables. If there is a better alternative, please let me know.
2.out of curious, why is it "logged" and "non-logged", not something like, "persisting" and "non-persisting"?
>> Not sure. We created a "LoggedStore" and not a "PersistantStore". That is probably why the terminology has stuck around.
> On May 6, 2015, 12:10 a.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala, lines 447-464
> > <https://reviews.apache.org/r/33453/diff/4/?file=950515#file950515line447>
> >
> > 1. why do we need defaultStoreBaseDire and LoggedStorageBaseDir? Can we just have one variable and one method, such that, storeBaseDir = getStoreBaseDir?
> >
> > 2. line 442 and line 454 are duplicated
> >
> > 3. from line 447-448, just call Util.getJobNameAndId to avoid duplicated code?
1. why do we need defaultStoreBaseDire and LoggedStorageBaseDir? Can we just have one variable and one method, such that, storeBaseDir = getStoreBaseDir?
>> In the proposed design (SAMZA-617), we want to periodically clean-up the data stores that are not being used. For stores that don't have to be persisted, it can be cleaned up as soon as the job exits. This is already done for us by YARN. If we choose to use the same location for both the stores, then we have to clean up after the job exits and we lose our ability to use YARN NM's deletionService. This just makes sure that we persist only the data stores that we need.
2. line 442 and line 454 are duplicated
>> For some reason, I wanted 2 different file object instances. I can't recollect why. I guess it can be avoided.
3. from line 447-448, just call Util.getJobNameAndId to avoid duplicated code?
>> Sure. That sounds good.
> On May 6, 2015, 12:10 a.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala, lines 481-486
> > <https://reviews.apache.org/r/33453/diff/4/?file=950515#file950515line481>
> >
> > 1. why does the changeLogSystemStreamPartition matter here?
> > 2. prefer the storePartitionDir, because it's the dir for each partition/task, right?
1. why does the changeLogSystemStreamPartition matter here?
>> changeLogSystemStreamPartition indicates whether a store has a changelog or not. Depending on the case, the base directory of the store will be different.
2. prefer the storePartitionDir, because it's the dir for each partition/task, right?
>> Base directory is same for all tasks in the job. There is a separate partition directory for each task within each store directory. The path looks like : "<StoreBaseDir>/<StoreName>/<TaskName>/<storeFiles>" . Does that clarify your question?
> On May 6, 2015, 12:10 a.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala, lines 508-509
> > <https://reviews.apache.org/r/33453/diff/4/?file=950515#file950515line508>
> >
> > why do we need 2 storeDir? Is it because we want to delete the default state Dir?
Yes
> On May 6, 2015, 12:10 a.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala, lines 110-112
> > <https://reviews.apache.org/r/33453/diff/4/?file=950516#file950516line110>
> >
> > personal opinion: should this logic be another method, such as readOffsetFromFile, not "cleanBaseDir"? A little misleading.
I can modularize it further if it helps. Since we need to load offsets (if available) before cleaning directories, it became a part of it.
Let me fix this.
- Navina
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/#review82610
-----------------------------------------------------------
On May 5, 2015, 6:47 p.m., Navina Ramesh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33453/
> -----------------------------------------------------------
>
> (Updated May 5, 2015, 6:47 p.m.)
>
>
> Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Added checksum to the Offset file and some unit tests
>
> Added Unit Tests for TaskStorageManager and refactored some code
>
> Changed default to yarn cwd instead of io.tmpDir and refactored code
>
>
> Diffs
> -----
>
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala e94a4735217f59d074510ce1556c8c439e6a72f0
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ac4793afe1e6868933e750181bee1e27c157b5e6
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 8a83566ae6139127d7fe04ab42231151227dc479
> samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala PRE-CREATION
> samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala b75f44060fb8e660e824eaeb9cfdcc9d6fa902e8
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 1b44a517129b35affac802929087eaa0061e6b5d
>
> Diff: https://reviews.apache.org/r/33453/diff/
>
>
> Testing
> -------
>
> Tested locally using hello-samza.
> Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
>
>
> Thanks,
>
> Navina Ramesh
>
>
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Yan Fang <ya...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/#review82610
-----------------------------------------------------------
samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala
<https://reviews.apache.org/r/33453/#comment133364>
1.is it also possible to retrived this value from the config? Then users do not need to set it in every node.
2.out of curious, why is it "logged" and "non-logged", not something like, "persisting" and "non-persisting"?
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala
<https://reviews.apache.org/r/33453/#comment133367>
1. why do we need defaultStoreBaseDire and LoggedStorageBaseDir? Can we just have one variable and one method, such that, storeBaseDir = getStoreBaseDir?
2. line 442 and line 454 are duplicated
3. from line 447-448, just call Util.getJobNameAndId to avoid duplicated code?
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala
<https://reviews.apache.org/r/33453/#comment133370>
1. why does the changeLogSystemStreamPartition matter here?
2. prefer the storePartitionDir, because it's the dir for each partition/task, right?
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala
<https://reviews.apache.org/r/33453/#comment133375>
why do we need 2 storeDir? Is it because we want to delete the default state Dir?
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala
<https://reviews.apache.org/r/33453/#comment133371>
personal opinion: should this logic be another method, such as readOffsetFromFile, not "cleanBaseDir"? A little misleading.
- Yan Fang
On May 5, 2015, 6:47 p.m., Navina Ramesh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33453/
> -----------------------------------------------------------
>
> (Updated May 5, 2015, 6:47 p.m.)
>
>
> Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Added checksum to the Offset file and some unit tests
>
> Added Unit Tests for TaskStorageManager and refactored some code
>
> Changed default to yarn cwd instead of io.tmpDir and refactored code
>
>
> Diffs
> -----
>
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala e94a4735217f59d074510ce1556c8c439e6a72f0
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ac4793afe1e6868933e750181bee1e27c157b5e6
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 8a83566ae6139127d7fe04ab42231151227dc479
> samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala PRE-CREATION
> samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala b75f44060fb8e660e824eaeb9cfdcc9d6fa902e8
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 1b44a517129b35affac802929087eaa0061e6b5d
>
> Diff: https://reviews.apache.org/r/33453/diff/
>
>
> Testing
> -------
>
> Tested locally using hello-samza.
> Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
>
>
> Thanks,
>
> Navina Ramesh
>
>
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Navina Ramesh <nr...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/
-----------------------------------------------------------
(Updated Sept. 21, 2015, 9:12 a.m.)
Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
Bugs: SAMZA-557
https://issues.apache.org/jira/browse/SAMZA-557
Repository: samza
Description
-------
Added checksum to the Offset file and some unit tests
Added Unit Tests for TaskStorageManager and refactored some code
Changed default to yarn cwd instead of io.tmpDir and refactored code
Diffs
-----
samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala e94a4735217f59d074510ce1556c8c439e6a72f0
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ac4793afe1e6868933e750181bee1e27c157b5e6
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
samza-core/src/main/scala/org/apache/samza/util/Util.scala 8a83566ae6139127d7fe04ab42231151227dc479
samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala PRE-CREATION
samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala b75f44060fb8e660e824eaeb9cfdcc9d6fa902e8
samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 1b44a517129b35affac802929087eaa0061e6b5d
Diff: https://reviews.apache.org/r/33453/diff/
Testing
-------
Tested locally using hello-samza.
Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
Thanks,
Navina Ramesh
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/#review82738
-----------------------------------------------------------
Overall looks good. Just one minor comment in the info log.
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala
<https://reviews.apache.org/r/33453/#comment133550>
Should be "default" here.
- Yi Pan (Data Infrastructure)
On May 6, 2015, 6:22 a.m., Navina Ramesh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33453/
> -----------------------------------------------------------
>
> (Updated May 6, 2015, 6:22 a.m.)
>
>
> Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Added checksum to the Offset file and some unit tests
>
> Added Unit Tests for TaskStorageManager and refactored some code
>
> Changed default to yarn cwd instead of io.tmpDir and refactored code
>
>
> Diffs
> -----
>
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala e94a4735217f59d074510ce1556c8c439e6a72f0
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ac4793afe1e6868933e750181bee1e27c157b5e6
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 8a83566ae6139127d7fe04ab42231151227dc479
> samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala PRE-CREATION
> samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala b75f44060fb8e660e824eaeb9cfdcc9d6fa902e8
> samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 1b44a517129b35affac802929087eaa0061e6b5d
>
> Diff: https://reviews.apache.org/r/33453/diff/
>
>
> Testing
> -------
>
> Tested locally using hello-samza.
> Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
>
>
> Thanks,
>
> Navina Ramesh
>
>
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Navina Ramesh <nr...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/
-----------------------------------------------------------
(Updated May 6, 2015, 6:22 a.m.)
Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
Repository: samza
Description
-------
Added checksum to the Offset file and some unit tests
Added Unit Tests for TaskStorageManager and refactored some code
Changed default to yarn cwd instead of io.tmpDir and refactored code
Diffs (updated)
-----
samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala e94a4735217f59d074510ce1556c8c439e6a72f0
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ac4793afe1e6868933e750181bee1e27c157b5e6
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
samza-core/src/main/scala/org/apache/samza/util/Util.scala 8a83566ae6139127d7fe04ab42231151227dc479
samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala PRE-CREATION
samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala b75f44060fb8e660e824eaeb9cfdcc9d6fa902e8
samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 1b44a517129b35affac802929087eaa0061e6b5d
Diff: https://reviews.apache.org/r/33453/diff/
Testing
-------
Tested locally using hello-samza.
Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
Thanks,
Navina Ramesh
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Navina Ramesh <nr...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/
-----------------------------------------------------------
(Updated May 5, 2015, 6:47 p.m.)
Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
Repository: samza
Description (updated)
-------
Added checksum to the Offset file and some unit tests
Added Unit Tests for TaskStorageManager and refactored some code
Changed default to yarn cwd instead of io.tmpDir and refactored code
Diffs (updated)
-----
samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala e94a4735217f59d074510ce1556c8c439e6a72f0
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ac4793afe1e6868933e750181bee1e27c157b5e6
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
samza-core/src/main/scala/org/apache/samza/util/Util.scala 8a83566ae6139127d7fe04ab42231151227dc479
samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala PRE-CREATION
samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala b75f44060fb8e660e824eaeb9cfdcc9d6fa902e8
samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 1b44a517129b35affac802929087eaa0061e6b5d
Diff: https://reviews.apache.org/r/33453/diff/
Testing
-------
Tested locally using hello-samza.
Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
Thanks,
Navina Ramesh
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Navina Ramesh <nr...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/
-----------------------------------------------------------
(Updated May 4, 2015, 8:57 p.m.)
Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
Repository: samza
Description (updated)
-------
Added Unit Tests for TaskStorageManager and refactored some code
Changed default to yarn cwd instead of io.tmpDir and refactored code
Diffs (updated)
-----
samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala e94a4735217f59d074510ce1556c8c439e6a72f0
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ac4793afe1e6868933e750181bee1e27c157b5e6
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala PRE-CREATION
samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 1b44a517129b35affac802929087eaa0061e6b5d
Diff: https://reviews.apache.org/r/33453/diff/
Testing
-------
Tested locally using hello-samza.
Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
Thanks,
Navina Ramesh
Re: Review Request 33453: SAMZA-557 Reuse local state in
SamzaContainer on clean shutdown
Posted by Navina Ramesh <nr...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33453/
-----------------------------------------------------------
(Updated April 22, 2015, 9:54 p.m.)
Review request for samza, Yan Fang, Chris Riccomini, Naveen Somasundaram, and Yi Pan (Data Infrastructure).
Changes
-------
Edited the documentation for the environment variable
Repository: samza
Description
-------
Changed default to yarn cwd instead of io.tmpDir and refactored code
Diffs (updated)
-----
samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 720fbdceafc4fe69b048d81a677e874d13e6d22f
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a
samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala 66c2a0dc2e38e21f951727a30f0987776ac52fe2
Diff: https://reviews.apache.org/r/33453/diff/
Testing
-------
Tested locally using hello-samza.
Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state.
Thanks,
Navina Ramesh