You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by amansinha100 <gi...@git.apache.org> on 2015/10/18 22:22:25 UTC
[GitHub] drill pull request: DRILL-3947: Use setSafe() for date, time, time...
GitHub user amansinha100 opened a pull request:
https://github.com/apache/drill/pull/208
DRILL-3947: Use setSafe() for date, time, timestamp types while popul…
…ating pruning vector (other types were already using setSafe).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/amansinha100/incubator-drill date1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/208.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #208
----
commit 7f0a03c17ca08433ebd2b0f122e23212b63c1570
Author: Aman Sinha <as...@maprtech.com>
Date: 2015-10-18T16:59:19Z
DRILL-3947: Use setSafe() for date, time, timestamp types while populating pruning vector (other types were already using setSafe).
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-3947: Use setSafe() for date, time, time...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/208
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-3947: Use setSafe() for date, time, time...
Posted by amansinha100 <gi...@git.apache.org>.
Github user amansinha100 commented on the pull request:
https://github.com/apache/drill/pull/208#issuecomment-149257300
@zfong, please see my last explanation about why the repro does not occur at small scale. Hence, adding a unit test won't help ... unless the static constant that controls the initial size of the value vectors is made configurable...but that would require more code changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-3947: Use setSafe() for date, time, time...
Posted by zfong <gi...@git.apache.org>.
Github user zfong commented on the pull request:
https://github.com/apache/drill/pull/208#issuecomment-149087340
Is there a small unit test case that reproduces this problem?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-3947: Use setSafe() for date, time, time...
Posted by zfong <gi...@git.apache.org>.
Github user zfong commented on the pull request:
https://github.com/apache/drill/pull/208#issuecomment-149259793
Aman -- yup, saw that. Thanks.
On Mon, Oct 19, 2015 at 8:57 AM, Aman Sinha <no...@github.com>
wrote:
> @zfong <https://github.com/zfong>, please see my last explanation about
> why the repro does not occur at small scale. Hence, adding a unit test
> won't help ... unless the static constant that controls the initial size of
> the value vectors is made configurable...but that would require more code
> changes.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/drill/pull/208#issuecomment-149257300>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-3947: Use setSafe() for date, time, time...
Posted by amansinha100 <gi...@git.apache.org>.
Github user amansinha100 commented on the pull request:
https://github.com/apache/drill/pull/208#issuecomment-149123146
@mehant , yes the underlying issue is simple: the null bit vector for the NullableDateVector is allocated at 4096 bytes (actually, the UInt1Vector is used for this, hence the error message shows length: 1). The 4096 is based on the static constant BaseValueVector.INITIAL_VALUE_ALLOCATION. Since set() does not do any reallocation, once this limit is reached, attempt to write to this vector causes IOBE. setSafe() does realloc()s. I am pretty sure I can create a test case with > 4096 files partitioned on date column that will repro this issue. I will need to work with QA/performance folks on creating this test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-3947: Use setSafe() for date, time, time...
Posted by mehant <gi...@git.apache.org>.
Github user mehant commented on the pull request:
https://github.com/apache/drill/pull/208#issuecomment-149087927
+1.
Changes look fine to me, given that all the other types use setSafe() it makes sense to use the same method for DATE, TIMESTAMP for consistency. However I feel that we might be addressing a symptom here, do you have any thoughts on what the actual issue might be?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---