You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Venki Korukanti <ve...@gmail.com> on 2015/02/12 02:48:57 UTC
Review Request 30910: DRILL-2178: Update outgoing record batch size
and allocation in PartitionSender
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30910/
-----------------------------------------------------------
Review request for drill and Steven Phillips.
Repository: drill-git
Description
-------
Please see the JIRA DRILL-2178 for details
Diffs
-----
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/materialize/QueryWritableBatch.java cef4101
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java f09acaa
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java 4292c09
exec/java-exec/src/main/java/org/apache/drill/exec/record/FragmentWritableBatch.java d122311
exec/java-exec/src/main/java/org/apache/drill/exec/record/MaterializedField.java a9f3292
Diff: https://reviews.apache.org/r/30910/diff/
Testing
-------
Thanks,
Venki Korukanti
Re: Review Request 30910: DRILL-2178: Update outgoing record batch
size and allocation in PartitionSender
Posted by Jacques Nadeau <ja...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30910/#review74281
-----------------------------------------------------------
Ship it!
Ship It!
- Jacques Nadeau
On Feb. 12, 2015, 9:52 p.m., Venki Korukanti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30910/
> -----------------------------------------------------------
>
> (Updated Feb. 12, 2015, 9:52 p.m.)
>
>
> Review request for drill and Steven Phillips.
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Please see the JIRA DRILL-2178 for details
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 52a3868
> exec/java-exec/src/main/codegen/templates/NullableValueVectors.java ba7c629
> exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java d39040e
> exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java aa5b702
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/materialize/QueryWritableBatch.java cef4101
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java f09acaa
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java 4292c09
> exec/java-exec/src/main/java/org/apache/drill/exec/record/FragmentWritableBatch.java d122311
> exec/java-exec/src/main/java/org/apache/drill/exec/record/MaterializedField.java bcc226f
> exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java f6644bd
> exec/java-exec/src/main/java/org/apache/drill/exec/vector/ObjectVector.java 3c15db3
> exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedVector.java b23ee02
> exec/java-exec/src/main/java/org/apache/drill/exec/vector/ValueVector.java df6a486
> exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java c5dc5ba
> exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java 3078f4e
> exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java 6dce363
> exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestValueVector.java 2bed433
>
> Diff: https://reviews.apache.org/r/30910/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Venki Korukanti
>
>
Re: Review Request 30910: DRILL-2178: Update outgoing record batch
size and allocation in PartitionSender
Posted by Venki Korukanti <ve...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30910/
-----------------------------------------------------------
(Updated Feb. 12, 2015, 9:52 p.m.)
Review request for drill and Steven Phillips.
Changes
-------
One issue observed is:
For vectors that have offsets (nullable, variable length), we allocate extra 1 record for offset vector. If you give 1024 as record capacity, offset vector is allocated for 1025 which may result in memory allocation for 2048 (next 2^x) number as netty tries to allocate in chunks of 2^x. So it is better to give initial capacity as number (2^x) - 1. This causes 1 record in values vector to be not utilized, but it is better than the memory wasted in offset vector when record capacity given is 2^x number.
Updating the PartitionSender to allocate 1023 or 511 records instead of earlier numbers 1024 and 512. Our current default vector allocation also has the similar problem. I will update it in next patch.
Repository: drill-git
Description
-------
Please see the JIRA DRILL-2178 for details
Diffs (updated)
-----
exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 52a3868
exec/java-exec/src/main/codegen/templates/NullableValueVectors.java ba7c629
exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java d39040e
exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java aa5b702
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/materialize/QueryWritableBatch.java cef4101
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java f09acaa
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java 4292c09
exec/java-exec/src/main/java/org/apache/drill/exec/record/FragmentWritableBatch.java d122311
exec/java-exec/src/main/java/org/apache/drill/exec/record/MaterializedField.java bcc226f
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java f6644bd
exec/java-exec/src/main/java/org/apache/drill/exec/vector/ObjectVector.java 3c15db3
exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedVector.java b23ee02
exec/java-exec/src/main/java/org/apache/drill/exec/vector/ValueVector.java df6a486
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java c5dc5ba
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java 3078f4e
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java 6dce363
exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestValueVector.java 2bed433
Diff: https://reviews.apache.org/r/30910/diff/
Testing
-------
Thanks,
Venki Korukanti
Re: Review Request 30910: DRILL-2178: Update outgoing record batch
size and allocation in PartitionSender
Posted by Venki Korukanti <ve...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30910/
-----------------------------------------------------------
(Updated Feb. 12, 2015, 5:34 p.m.)
Review request for drill and Steven Phillips.
Repository: drill-git
Description
-------
Please see the JIRA DRILL-2178 for details
Diffs (updated)
-----
exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 52a3868
exec/java-exec/src/main/codegen/templates/NullableValueVectors.java ba7c629
exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java d39040e
exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java aa5b702
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/materialize/QueryWritableBatch.java cef4101
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java f09acaa
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java 4292c09
exec/java-exec/src/main/java/org/apache/drill/exec/record/FragmentWritableBatch.java d122311
exec/java-exec/src/main/java/org/apache/drill/exec/record/MaterializedField.java bcc226f
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java f6644bd
exec/java-exec/src/main/java/org/apache/drill/exec/vector/ObjectVector.java 3c15db3
exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedVector.java b23ee02
exec/java-exec/src/main/java/org/apache/drill/exec/vector/ValueVector.java df6a486
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java c5dc5ba
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java 3078f4e
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java 6dce363
exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestValueVector.java 2bed433
Diff: https://reviews.apache.org/r/30910/diff/
Testing
-------
Thanks,
Venki Korukanti
Re: Review Request 30910: DRILL-2178: Update outgoing record batch
size and allocation in PartitionSender
Posted by Venki Korukanti <ve...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30910/
-----------------------------------------------------------
(Updated Feb. 12, 2015, 5:21 p.m.)
Review request for drill and Steven Phillips.
Changes
-------
Add a method to set initial capacity of ValueVectors. Initial memory allocated using allocateNew() is sufficient to fit the given initial capacity.
Repository: drill-git
Description
-------
Please see the JIRA DRILL-2178 for details
Diffs (updated)
-----
exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 52a3868
exec/java-exec/src/main/codegen/templates/NullableValueVectors.java ba7c629
exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java d39040e
exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java aa5b702
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/materialize/QueryWritableBatch.java cef4101
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java f09acaa
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java 4292c09
exec/java-exec/src/main/java/org/apache/drill/exec/record/FragmentWritableBatch.java d122311
exec/java-exec/src/main/java/org/apache/drill/exec/record/MaterializedField.java bcc226f
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java f6644bd
exec/java-exec/src/main/java/org/apache/drill/exec/vector/ObjectVector.java 3c15db3
exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedVector.java b23ee02
exec/java-exec/src/main/java/org/apache/drill/exec/vector/ValueVector.java df6a486
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java c5dc5ba
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java 3078f4e
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java 6dce363
exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestValueVector.java 2bed433
Diff: https://reviews.apache.org/r/30910/diff/
Testing
-------
Thanks,
Venki Korukanti