You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by Chris McCubbin <ch...@sqrrl.com> on 2013/11/25 22:37:05 UTC
Re: Review Request 15846: Prototype implementation of ACCUMULO-1787.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15846/
-----------------------------------------------------------
(Updated Nov. 25, 2013, 9:37 p.m.)
Review request for accumulo, Adam Fuchs, Josh Elser, and kturner.
Changes
-------
added accumulo group
Repository: accumulo
Description
-------
This is a prototype implementation of ACCUMULO-1787.
Diffs
-----
core/src/main/java/org/apache/accumulo/core/conf/Property.java f4c1778
server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java 8e4af64
Diff: https://reviews.apache.org/r/15846/diff/
Testing
-------
Thanks,
Chris McCubbin
Re: Review Request 15846: Prototype implementation of ACCUMULO-1787.
Posted by Chris McCubbin <ch...@sqrrl.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15846/
-----------------------------------------------------------
(Updated Nov. 24, 2014, 1:31 p.m.)
Review request for accumulo, Adam Fuchs, Josh Elser, and kturner.
Changes
-------
Link to the JIRA issue.
Bugs: ACCUMULO-1787
https://issues.apache.org/jira/browse/ACCUMULO-1787
Repository: accumulo
Description
-------
This is a prototype implementation of ACCUMULO-1787.
Diffs
-----
core/src/main/java/org/apache/accumulo/core/conf/Property.java f4c1778
server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java 8e4af64
Diff: https://reviews.apache.org/r/15846/diff/
Testing
-------
Thanks,
Chris McCubbin
Re: Review Request 15846: Prototype implementation of ACCUMULO-1787.
Posted by Chris McCubbin <ch...@sqrrl.com>.
> On Nov. 26, 2013, 2:32 p.m., kturner wrote:
> > server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java, line 300
> > <https://reviews.apache.org/r/15846/diff/2/?file=391121#file391121line300>
> >
> > The following situation is kinda bugging me, but I do not think its very important.
> >
> > 1. some snappy files exceed threshold
> > 2. creates a gzip file thats below threshold
> > 3. forced compaction of that gzip file creates a snappy file
> >
> >
> >
> >
I had actually thought about this as a potential issue as well. Can it actually happen though? Can the gzip file below the threshold be compacted again without combining with another file?
- Chris
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15846/#review29433
-----------------------------------------------------------
On Nov. 26, 2013, 4:32 a.m., Chris McCubbin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15846/
> -----------------------------------------------------------
>
> (Updated Nov. 26, 2013, 4:32 a.m.)
>
>
> Review request for accumulo, Adam Fuchs, Josh Elser, and kturner.
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> This is a prototype implementation of ACCUMULO-1787.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/accumulo/core/conf/Property.java f4c1778
> server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java 8e4af64
>
> Diff: https://reviews.apache.org/r/15846/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Chris McCubbin
>
>
Re: Review Request 15846: Prototype implementation of ACCUMULO-1787.
Posted by Josh Elser <jo...@gmail.com>.
> On Nov. 26, 2013, 2:32 p.m., kturner wrote:
> > server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java, line 296
> > <https://reviews.apache.org/r/15846/diff/2/?file=391121#file391121line296>
> >
> > Bulk imported files may not have an estimated size. I can not remember, something to look into.
I believe you're correct, Keith.
- Josh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15846/#review29433
-----------------------------------------------------------
On Nov. 26, 2013, 4:32 a.m., Chris McCubbin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15846/
> -----------------------------------------------------------
>
> (Updated Nov. 26, 2013, 4:32 a.m.)
>
>
> Review request for accumulo, Adam Fuchs, Josh Elser, and kturner.
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> This is a prototype implementation of ACCUMULO-1787.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/accumulo/core/conf/Property.java f4c1778
> server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java 8e4af64
>
> Diff: https://reviews.apache.org/r/15846/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Chris McCubbin
>
>
Re: Review Request 15846: Prototype implementation of ACCUMULO-1787.
Posted by ke...@deenlo.com.
> On Nov. 26, 2013, 2:32 p.m., kturner wrote:
> > server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java, line 300
> > <https://reviews.apache.org/r/15846/diff/2/?file=391121#file391121line300>
> >
> > The following situation is kinda bugging me, but I do not think its very important.
> >
> > 1. some snappy files exceed threshold
> > 2. creates a gzip file thats below threshold
> > 3. forced compaction of that gzip file creates a snappy file
> >
> >
> >
> >
>
> Chris McCubbin wrote:
> I had actually thought about this as a potential issue as well. Can it actually happen though? Can the gzip file below the threshold be compacted again without combining with another file?
A user initiated compaction will compact all files in a tablet, even if there is only one file. This is useful for applying filters, etc.
- kturner
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15846/#review29433
-----------------------------------------------------------
On Nov. 26, 2013, 4:32 a.m., Chris McCubbin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15846/
> -----------------------------------------------------------
>
> (Updated Nov. 26, 2013, 4:32 a.m.)
>
>
> Review request for accumulo, Adam Fuchs, Josh Elser, and kturner.
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> This is a prototype implementation of ACCUMULO-1787.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/accumulo/core/conf/Property.java f4c1778
> server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java 8e4af64
>
> Diff: https://reviews.apache.org/r/15846/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Chris McCubbin
>
>
Re: Review Request 15846: Prototype implementation of ACCUMULO-1787.
Posted by ke...@deenlo.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15846/#review29433
-----------------------------------------------------------
server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java
<https://reviews.apache.org/r/15846/#comment56649>
Bulk imported files may not have an estimated size. I can not remember, something to look into.
server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java
<https://reviews.apache.org/r/15846/#comment56650>
The following situation is kinda bugging me, but I do not think its very important.
1. some snappy files exceed threshold
2. creates a gzip file thats below threshold
3. forced compaction of that gzip file creates a snappy file
- kturner
On Nov. 26, 2013, 4:32 a.m., Chris McCubbin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15846/
> -----------------------------------------------------------
>
> (Updated Nov. 26, 2013, 4:32 a.m.)
>
>
> Review request for accumulo, Adam Fuchs, Josh Elser, and kturner.
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> This is a prototype implementation of ACCUMULO-1787.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/accumulo/core/conf/Property.java f4c1778
> server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java 8e4af64
>
> Diff: https://reviews.apache.org/r/15846/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Chris McCubbin
>
>
Re: Review Request 15846: Prototype implementation of ACCUMULO-1787.
Posted by Chris McCubbin <ch...@sqrrl.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15846/
-----------------------------------------------------------
(Updated Nov. 26, 2013, 4:32 a.m.)
Review request for accumulo, Adam Fuchs, Josh Elser, and kturner.
Changes
-------
Patch updated with suggestions. Moved to a small file threshold so the default behavior is the same as before.
Repository: accumulo
Description
-------
This is a prototype implementation of ACCUMULO-1787.
Diffs (updated)
-----
core/src/main/java/org/apache/accumulo/core/conf/Property.java f4c1778
server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java 8e4af64
Diff: https://reviews.apache.org/r/15846/diff/
Testing
-------
Thanks,
Chris McCubbin
Re: Review Request 15846: Prototype implementation of ACCUMULO-1787.
Posted by Adam Fuchs <af...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15846/#review29395
-----------------------------------------------------------
core/src/main/java/org/apache/accumulo/core/conf/Property.java
<https://reviews.apache.org/r/15846/#comment56597>
Whitespace changes should be removed.
core/src/main/java/org/apache/accumulo/core/conf/Property.java
<https://reviews.apache.org/r/15846/#comment56599>
Instead of specifying the compression threshold as a count in "millions of bytes" it should use PropertyType.MEMORY.
core/src/main/java/org/apache/accumulo/core/conf/Property.java
<https://reviews.apache.org/r/15846/#comment56603>
We need a way to determine whether this option has been enabled, and configuration should default to the previous behavior if this is not enabled. This could be done with a file compression threshold of 0B, and then the alternative compression type would be for small files instead of large files.
server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java
<https://reviews.apache.org/r/15846/#comment56606>
This should only be done if the alternative compression type is enabled. If we default the threshold to 0B and flip the > to a < that would be sufficient.
- Adam Fuchs
On Nov. 25, 2013, 9:37 p.m., Chris McCubbin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15846/
> -----------------------------------------------------------
>
> (Updated Nov. 25, 2013, 9:37 p.m.)
>
>
> Review request for accumulo, Adam Fuchs, Josh Elser, and kturner.
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> This is a prototype implementation of ACCUMULO-1787.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/accumulo/core/conf/Property.java f4c1778
> server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java 8e4af64
>
> Diff: https://reviews.apache.org/r/15846/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Chris McCubbin
>
>
Re: Review Request 15846: Prototype implementation of ACCUMULO-1787.
Posted by Eric Newton <er...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15846/#review29396
-----------------------------------------------------------
core/src/main/java/org/apache/accumulo/core/conf/Property.java
<https://reviews.apache.org/r/15846/#comment56598>
Use a default of "50M"... it will be consistent with other size properties. Should use the PropertyType.MEMORY.
server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java
<https://reviews.apache.org/r/15846/#comment56600>
Just use acuTableConf.getMemoryInBytes, which will handle magical "M" and "G" suffixes.
- Eric Newton
On Nov. 25, 2013, 9:37 p.m., Chris McCubbin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15846/
> -----------------------------------------------------------
>
> (Updated Nov. 25, 2013, 9:37 p.m.)
>
>
> Review request for accumulo, Adam Fuchs, Josh Elser, and kturner.
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> This is a prototype implementation of ACCUMULO-1787.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/accumulo/core/conf/Property.java f4c1778
> server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java 8e4af64
>
> Diff: https://reviews.apache.org/r/15846/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Chris McCubbin
>
>