You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by dong wang <el...@gmail.com> on 2015/03/16 12:11:33 UTC
error when refreshing the latest segment
The latest segment is the newest one which contains today's data, due to
some incorrect data, I want to refresh the segment, however, when
cube->actions->refresh->select the latest segment, then the following error
occurs:
New segment range invalid, start date must be earlier than end date,
1426464000000 < 1426464000000
Re: error when refreshing the latest segment
Posted by dong wang <el...@gmail.com>.
as confirmed, the issue is fixed by your updates.
2015-03-17 11:22 GMT+08:00 Shi, Shaofeng <sh...@ebay.com>:
> I made a fix on the staging branch, the binary package on download page
> need sometime to be updated; if you couldn’t wait, please pull the latest
> code, checkout staging branch, and then run ./script/package.sh to build a
> new package; Please let me know if still have problem; this is the JIRA
> for this problem:
>
> https://issues.apache.org/jira/browse/KYLIN-642
>
>
> On 3/17/15, 9:59 AM, "dong wang" <el...@gmail.com> wrote:
>
> >many thanks, the issue may not that be difficult but very very urgent
> >since it affects the rollback feature
> >
> >2015-03-17 9:50 GMT+08:00 Shi, Shaofeng <sh...@ebay.com>:
> >
> >> Okay, I was able to reproduce this problem; will look into it now;
> >>
> >> On 3/17/15, 9:45 AM, "dong wang" <el...@gmail.com> wrote:
> >>
> >> >Shaofeng, I think mine is a little different from what you mentioned:
> >> >1, the partition date column can be recognized correctly
> >> >2, the segment is built successfully already
> >> >3, I just refresh the LATEST successful segment due to some base data
> >> >changes
> >> >
> >> >2015-03-16 22:03 GMT+08:00 Shi, Shaofeng <sh...@ebay.com>:
> >> >
> >> >> Hi Dong, which version of the Kylin build that you deployed?
> >> >>
> >> >> In last week we found a bug which will cause Kylin fail to allocate a
> >> >>new
> >> >> a segment; Kylin will report error saying that the startDate is not
> >> >> matched with expectation; Here is the JIRA:
> >> >>
> >> >> https://issues.apache.org/jira/browse/KYLIN-628
> >> >>
> >> >> If you¹re using Kylin¹s binary package, please update to the latest
> >>one
> >> >>on
> >> >> https://kylin.incubator.apache.org/download/
> >> >> If you¹re running from source code, please pull the latest code on
> >> >>staging
> >> >> branch;
> >> >>
> >> >> On 3/16/15, 7:40 PM, "dong wang" <el...@gmail.com> wrote:
> >> >>
> >> >> >since the following codes pass invalid startdate and enddate, which
> >> >>leads
> >> >> >to the issue
> >> >> >
> >> >> > public CubeSegment appendSegments(CubeInstance cube, long endDate)
> >> >> >throws
> >> >> >IOException {
> >> >> > checkNoBuildingSegment(cube);
> >> >> >
> >> >> > CubeSegment newSegment;
> >> >> > if
> >> >>
> >>>(cube.getDescriptor().getModel().getPartitionDesc().isPartitioned()) {
> >> >> > long startDate =
> >>calculateStartDateForAppendSegment(cube);
> >> >> > newSegment = newSegment(cube, startDate, endDate);
> >> >> > } else {
> >> >> > newSegment = newSegment(cube, 0, Long.MAX_VALUE);
> >> >> > }
> >> >> >
> >> >> > validateNewSegments(cube, newSegment);
> >> >> > cube.getSegments().add(newSegment);
> >> >> > Collections.sort(cube.getSegments());
> >> >> > updateCube(cube);
> >> >> >
> >> >> > return newSegment;
> >> >> > }
> >> >> >
> >> >> >2015-03-16 19:38 GMT+08:00 dong wang <el...@gmail.com>:
> >> >> >
> >> >> >> The related codes, since I'm not that clear about the logic it
> >>should
> >> >> >>be:
> >> >> >>
> >> >> >> private CubeSegment newSegment(CubeInstance cubeInstance, long
> >> >> >> startDate, long endDate) {
> >> >> >> if (startDate >= endDate)
> >> >> >> throw new IllegalArgumentException("New segment range
> >> >> >>invalid,
> >> >> >> start date must be earlier than end date, " + startDate + " < " +
> >> >> >>endDate);
> >> >> >>
> >> >> >> CubeSegment segment = new CubeSegment();
> >> >> >> String incrementalSegName =
> >> >> >>CubeSegment.getSegmentName(startDate,
> >> >> >> endDate);
> >> >> >> segment.setUuid(UUID.randomUUID().toString());
> >> >> >> segment.setName(incrementalSegName);
> >> >> >> Date creatTime = new Date();
> >> >> >> segment.setCreateTimeUTC(creatTime.getTime());
> >> >> >> segment.setDateRangeStart(startDate);
> >> >> >> segment.setDateRangeEnd(endDate);
> >> >> >> segment.setStatus(SegmentStatusEnum.NEW);
> >> >> >>
> >> >>segment.setStorageLocationIdentifier(generateStorageLocation());
> >> >> >>
> >> >> >> segment.setCubeInstance(cubeInstance);
> >> >> >>
> >> >> >> segment.validate();
> >> >> >> return segment;
> >> >> >> }
> >> >> >>
> >> >> >> 2015-03-16 19:29 GMT+08:00 dong wang <el...@gmail.com>:
> >> >> >>
> >> >> >>> more notes. this segment to be refreshed is built sccessful
> >>first,
> >> >> >>>then,
> >> >> >>> I plan to refresh it due to some incorrect data.
> >> >> >>>
> >> >> >>> 2015-03-16 19:25 GMT+08:00 dong wang <el...@gmail.com>:
> >> >> >>>
> >> >> >>>> watched with firebug, the parameters pass to the backend seem
> >>to be
> >> >> >>>> CORRECT as below:
> >> >> >>>>
> >> >> >>>> {buildType: "BUILD", startTime: 1426377600000, endTime:
> >> >>1426464000000}
> >> >> >>>>
> >> >> >>>> 2015-03-16 19:15 GMT+08:00 dong wang <el...@gmail.com>:
> >> >> >>>>
> >> >> >>>>> segment detail page shows:
> >> >> >>>>>
> >> >> >>>>> Start Date (Include) 2015-03-15 00:00:00
> >> >> >>>>> End Date (Exclude) 2015-03-16 00:00:00
> >> >> >>>>>
> >> >> >>>>> 2015-03-16 19:11 GMT+08:00 dong wang <el...@gmail.com>:
> >> >> >>>>>
> >> >> >>>>>> The latest segment is the newest one which contains today's
> >>data,
> >> >> >>>>>>due
> >> >> >>>>>> to some incorrect data, I want to refresh the segment,
> >>however,
> >> >>when
> >> >> >>>>>> cube->actions->refresh->select the latest segment, then the
> >> >> >>>>>>following error
> >> >> >>>>>> occurs:
> >> >> >>>>>>
> >> >> >>>>>> New segment range invalid, start date must be earlier than end
> >> >>date,
> >> >> >>>>>> 1426464000000 < 1426464000000
> >> >> >>>>>>
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>
> >> >> >>>
> >> >> >>
> >> >>
> >> >>
> >>
> >>
>
>
Re: error when refreshing the latest segment
Posted by "Shi, Shaofeng" <sh...@ebay.com>.
I made a fix on the staging branch, the binary package on download page
need sometime to be updated; if you couldn’t wait, please pull the latest
code, checkout staging branch, and then run ./script/package.sh to build a
new package; Please let me know if still have problem; this is the JIRA
for this problem:
https://issues.apache.org/jira/browse/KYLIN-642
On 3/17/15, 9:59 AM, "dong wang" <el...@gmail.com> wrote:
>many thanks, the issue may not that be difficult but very very urgent
>since it affects the rollback feature
>
>2015-03-17 9:50 GMT+08:00 Shi, Shaofeng <sh...@ebay.com>:
>
>> Okay, I was able to reproduce this problem; will look into it now;
>>
>> On 3/17/15, 9:45 AM, "dong wang" <el...@gmail.com> wrote:
>>
>> >Shaofeng, I think mine is a little different from what you mentioned:
>> >1, the partition date column can be recognized correctly
>> >2, the segment is built successfully already
>> >3, I just refresh the LATEST successful segment due to some base data
>> >changes
>> >
>> >2015-03-16 22:03 GMT+08:00 Shi, Shaofeng <sh...@ebay.com>:
>> >
>> >> Hi Dong, which version of the Kylin build that you deployed?
>> >>
>> >> In last week we found a bug which will cause Kylin fail to allocate a
>> >>new
>> >> a segment; Kylin will report error saying that the startDate is not
>> >> matched with expectation; Here is the JIRA:
>> >>
>> >> https://issues.apache.org/jira/browse/KYLIN-628
>> >>
>> >> If you¹re using Kylin¹s binary package, please update to the latest
>>one
>> >>on
>> >> https://kylin.incubator.apache.org/download/
>> >> If you¹re running from source code, please pull the latest code on
>> >>staging
>> >> branch;
>> >>
>> >> On 3/16/15, 7:40 PM, "dong wang" <el...@gmail.com> wrote:
>> >>
>> >> >since the following codes pass invalid startdate and enddate, which
>> >>leads
>> >> >to the issue
>> >> >
>> >> > public CubeSegment appendSegments(CubeInstance cube, long endDate)
>> >> >throws
>> >> >IOException {
>> >> > checkNoBuildingSegment(cube);
>> >> >
>> >> > CubeSegment newSegment;
>> >> > if
>> >>
>>>(cube.getDescriptor().getModel().getPartitionDesc().isPartitioned()) {
>> >> > long startDate =
>>calculateStartDateForAppendSegment(cube);
>> >> > newSegment = newSegment(cube, startDate, endDate);
>> >> > } else {
>> >> > newSegment = newSegment(cube, 0, Long.MAX_VALUE);
>> >> > }
>> >> >
>> >> > validateNewSegments(cube, newSegment);
>> >> > cube.getSegments().add(newSegment);
>> >> > Collections.sort(cube.getSegments());
>> >> > updateCube(cube);
>> >> >
>> >> > return newSegment;
>> >> > }
>> >> >
>> >> >2015-03-16 19:38 GMT+08:00 dong wang <el...@gmail.com>:
>> >> >
>> >> >> The related codes, since I'm not that clear about the logic it
>>should
>> >> >>be:
>> >> >>
>> >> >> private CubeSegment newSegment(CubeInstance cubeInstance, long
>> >> >> startDate, long endDate) {
>> >> >> if (startDate >= endDate)
>> >> >> throw new IllegalArgumentException("New segment range
>> >> >>invalid,
>> >> >> start date must be earlier than end date, " + startDate + " < " +
>> >> >>endDate);
>> >> >>
>> >> >> CubeSegment segment = new CubeSegment();
>> >> >> String incrementalSegName =
>> >> >>CubeSegment.getSegmentName(startDate,
>> >> >> endDate);
>> >> >> segment.setUuid(UUID.randomUUID().toString());
>> >> >> segment.setName(incrementalSegName);
>> >> >> Date creatTime = new Date();
>> >> >> segment.setCreateTimeUTC(creatTime.getTime());
>> >> >> segment.setDateRangeStart(startDate);
>> >> >> segment.setDateRangeEnd(endDate);
>> >> >> segment.setStatus(SegmentStatusEnum.NEW);
>> >> >>
>> >>segment.setStorageLocationIdentifier(generateStorageLocation());
>> >> >>
>> >> >> segment.setCubeInstance(cubeInstance);
>> >> >>
>> >> >> segment.validate();
>> >> >> return segment;
>> >> >> }
>> >> >>
>> >> >> 2015-03-16 19:29 GMT+08:00 dong wang <el...@gmail.com>:
>> >> >>
>> >> >>> more notes. this segment to be refreshed is built sccessful
>>first,
>> >> >>>then,
>> >> >>> I plan to refresh it due to some incorrect data.
>> >> >>>
>> >> >>> 2015-03-16 19:25 GMT+08:00 dong wang <el...@gmail.com>:
>> >> >>>
>> >> >>>> watched with firebug, the parameters pass to the backend seem
>>to be
>> >> >>>> CORRECT as below:
>> >> >>>>
>> >> >>>> {buildType: "BUILD", startTime: 1426377600000, endTime:
>> >>1426464000000}
>> >> >>>>
>> >> >>>> 2015-03-16 19:15 GMT+08:00 dong wang <el...@gmail.com>:
>> >> >>>>
>> >> >>>>> segment detail page shows:
>> >> >>>>>
>> >> >>>>> Start Date (Include) 2015-03-15 00:00:00
>> >> >>>>> End Date (Exclude) 2015-03-16 00:00:00
>> >> >>>>>
>> >> >>>>> 2015-03-16 19:11 GMT+08:00 dong wang <el...@gmail.com>:
>> >> >>>>>
>> >> >>>>>> The latest segment is the newest one which contains today's
>>data,
>> >> >>>>>>due
>> >> >>>>>> to some incorrect data, I want to refresh the segment,
>>however,
>> >>when
>> >> >>>>>> cube->actions->refresh->select the latest segment, then the
>> >> >>>>>>following error
>> >> >>>>>> occurs:
>> >> >>>>>>
>> >> >>>>>> New segment range invalid, start date must be earlier than end
>> >>date,
>> >> >>>>>> 1426464000000 < 1426464000000
>> >> >>>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>
>> >> >>>
>> >> >>
>> >>
>> >>
>>
>>
Re: error when refreshing the latest segment
Posted by dong wang <el...@gmail.com>.
many thanks, the issue may not that be difficult but very very urgent
since it affects the rollback feature
2015-03-17 9:50 GMT+08:00 Shi, Shaofeng <sh...@ebay.com>:
> Okay, I was able to reproduce this problem; will look into it now;
>
> On 3/17/15, 9:45 AM, "dong wang" <el...@gmail.com> wrote:
>
> >Shaofeng, I think mine is a little different from what you mentioned:
> >1, the partition date column can be recognized correctly
> >2, the segment is built successfully already
> >3, I just refresh the LATEST successful segment due to some base data
> >changes
> >
> >2015-03-16 22:03 GMT+08:00 Shi, Shaofeng <sh...@ebay.com>:
> >
> >> Hi Dong, which version of the Kylin build that you deployed?
> >>
> >> In last week we found a bug which will cause Kylin fail to allocate a
> >>new
> >> a segment; Kylin will report error saying that the startDate is not
> >> matched with expectation; Here is the JIRA:
> >>
> >> https://issues.apache.org/jira/browse/KYLIN-628
> >>
> >> If you¹re using Kylin¹s binary package, please update to the latest one
> >>on
> >> https://kylin.incubator.apache.org/download/
> >> If you¹re running from source code, please pull the latest code on
> >>staging
> >> branch;
> >>
> >> On 3/16/15, 7:40 PM, "dong wang" <el...@gmail.com> wrote:
> >>
> >> >since the following codes pass invalid startdate and enddate, which
> >>leads
> >> >to the issue
> >> >
> >> > public CubeSegment appendSegments(CubeInstance cube, long endDate)
> >> >throws
> >> >IOException {
> >> > checkNoBuildingSegment(cube);
> >> >
> >> > CubeSegment newSegment;
> >> > if
> >> >(cube.getDescriptor().getModel().getPartitionDesc().isPartitioned()) {
> >> > long startDate = calculateStartDateForAppendSegment(cube);
> >> > newSegment = newSegment(cube, startDate, endDate);
> >> > } else {
> >> > newSegment = newSegment(cube, 0, Long.MAX_VALUE);
> >> > }
> >> >
> >> > validateNewSegments(cube, newSegment);
> >> > cube.getSegments().add(newSegment);
> >> > Collections.sort(cube.getSegments());
> >> > updateCube(cube);
> >> >
> >> > return newSegment;
> >> > }
> >> >
> >> >2015-03-16 19:38 GMT+08:00 dong wang <el...@gmail.com>:
> >> >
> >> >> The related codes, since I'm not that clear about the logic it should
> >> >>be:
> >> >>
> >> >> private CubeSegment newSegment(CubeInstance cubeInstance, long
> >> >> startDate, long endDate) {
> >> >> if (startDate >= endDate)
> >> >> throw new IllegalArgumentException("New segment range
> >> >>invalid,
> >> >> start date must be earlier than end date, " + startDate + " < " +
> >> >>endDate);
> >> >>
> >> >> CubeSegment segment = new CubeSegment();
> >> >> String incrementalSegName =
> >> >>CubeSegment.getSegmentName(startDate,
> >> >> endDate);
> >> >> segment.setUuid(UUID.randomUUID().toString());
> >> >> segment.setName(incrementalSegName);
> >> >> Date creatTime = new Date();
> >> >> segment.setCreateTimeUTC(creatTime.getTime());
> >> >> segment.setDateRangeStart(startDate);
> >> >> segment.setDateRangeEnd(endDate);
> >> >> segment.setStatus(SegmentStatusEnum.NEW);
> >> >>
> >>segment.setStorageLocationIdentifier(generateStorageLocation());
> >> >>
> >> >> segment.setCubeInstance(cubeInstance);
> >> >>
> >> >> segment.validate();
> >> >> return segment;
> >> >> }
> >> >>
> >> >> 2015-03-16 19:29 GMT+08:00 dong wang <el...@gmail.com>:
> >> >>
> >> >>> more notes. this segment to be refreshed is built sccessful first,
> >> >>>then,
> >> >>> I plan to refresh it due to some incorrect data.
> >> >>>
> >> >>> 2015-03-16 19:25 GMT+08:00 dong wang <el...@gmail.com>:
> >> >>>
> >> >>>> watched with firebug, the parameters pass to the backend seem to be
> >> >>>> CORRECT as below:
> >> >>>>
> >> >>>> {buildType: "BUILD", startTime: 1426377600000, endTime:
> >>1426464000000}
> >> >>>>
> >> >>>> 2015-03-16 19:15 GMT+08:00 dong wang <el...@gmail.com>:
> >> >>>>
> >> >>>>> segment detail page shows:
> >> >>>>>
> >> >>>>> Start Date (Include) 2015-03-15 00:00:00
> >> >>>>> End Date (Exclude) 2015-03-16 00:00:00
> >> >>>>>
> >> >>>>> 2015-03-16 19:11 GMT+08:00 dong wang <el...@gmail.com>:
> >> >>>>>
> >> >>>>>> The latest segment is the newest one which contains today's data,
> >> >>>>>>due
> >> >>>>>> to some incorrect data, I want to refresh the segment, however,
> >>when
> >> >>>>>> cube->actions->refresh->select the latest segment, then the
> >> >>>>>>following error
> >> >>>>>> occurs:
> >> >>>>>>
> >> >>>>>> New segment range invalid, start date must be earlier than end
> >>date,
> >> >>>>>> 1426464000000 < 1426464000000
> >> >>>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> >>
>
>
Re: error when refreshing the latest segment
Posted by "Shi, Shaofeng" <sh...@ebay.com>.
Okay, I was able to reproduce this problem; will look into it now;
On 3/17/15, 9:45 AM, "dong wang" <el...@gmail.com> wrote:
>Shaofeng, I think mine is a little different from what you mentioned:
>1, the partition date column can be recognized correctly
>2, the segment is built successfully already
>3, I just refresh the LATEST successful segment due to some base data
>changes
>
>2015-03-16 22:03 GMT+08:00 Shi, Shaofeng <sh...@ebay.com>:
>
>> Hi Dong, which version of the Kylin build that you deployed?
>>
>> In last week we found a bug which will cause Kylin fail to allocate a
>>new
>> a segment; Kylin will report error saying that the startDate is not
>> matched with expectation; Here is the JIRA:
>>
>> https://issues.apache.org/jira/browse/KYLIN-628
>>
>> If you¹re using Kylin¹s binary package, please update to the latest one
>>on
>> https://kylin.incubator.apache.org/download/
>> If you¹re running from source code, please pull the latest code on
>>staging
>> branch;
>>
>> On 3/16/15, 7:40 PM, "dong wang" <el...@gmail.com> wrote:
>>
>> >since the following codes pass invalid startdate and enddate, which
>>leads
>> >to the issue
>> >
>> > public CubeSegment appendSegments(CubeInstance cube, long endDate)
>> >throws
>> >IOException {
>> > checkNoBuildingSegment(cube);
>> >
>> > CubeSegment newSegment;
>> > if
>> >(cube.getDescriptor().getModel().getPartitionDesc().isPartitioned()) {
>> > long startDate = calculateStartDateForAppendSegment(cube);
>> > newSegment = newSegment(cube, startDate, endDate);
>> > } else {
>> > newSegment = newSegment(cube, 0, Long.MAX_VALUE);
>> > }
>> >
>> > validateNewSegments(cube, newSegment);
>> > cube.getSegments().add(newSegment);
>> > Collections.sort(cube.getSegments());
>> > updateCube(cube);
>> >
>> > return newSegment;
>> > }
>> >
>> >2015-03-16 19:38 GMT+08:00 dong wang <el...@gmail.com>:
>> >
>> >> The related codes, since I'm not that clear about the logic it should
>> >>be:
>> >>
>> >> private CubeSegment newSegment(CubeInstance cubeInstance, long
>> >> startDate, long endDate) {
>> >> if (startDate >= endDate)
>> >> throw new IllegalArgumentException("New segment range
>> >>invalid,
>> >> start date must be earlier than end date, " + startDate + " < " +
>> >>endDate);
>> >>
>> >> CubeSegment segment = new CubeSegment();
>> >> String incrementalSegName =
>> >>CubeSegment.getSegmentName(startDate,
>> >> endDate);
>> >> segment.setUuid(UUID.randomUUID().toString());
>> >> segment.setName(incrementalSegName);
>> >> Date creatTime = new Date();
>> >> segment.setCreateTimeUTC(creatTime.getTime());
>> >> segment.setDateRangeStart(startDate);
>> >> segment.setDateRangeEnd(endDate);
>> >> segment.setStatus(SegmentStatusEnum.NEW);
>> >>
>>segment.setStorageLocationIdentifier(generateStorageLocation());
>> >>
>> >> segment.setCubeInstance(cubeInstance);
>> >>
>> >> segment.validate();
>> >> return segment;
>> >> }
>> >>
>> >> 2015-03-16 19:29 GMT+08:00 dong wang <el...@gmail.com>:
>> >>
>> >>> more notes. this segment to be refreshed is built sccessful first,
>> >>>then,
>> >>> I plan to refresh it due to some incorrect data.
>> >>>
>> >>> 2015-03-16 19:25 GMT+08:00 dong wang <el...@gmail.com>:
>> >>>
>> >>>> watched with firebug, the parameters pass to the backend seem to be
>> >>>> CORRECT as below:
>> >>>>
>> >>>> {buildType: "BUILD", startTime: 1426377600000, endTime:
>>1426464000000}
>> >>>>
>> >>>> 2015-03-16 19:15 GMT+08:00 dong wang <el...@gmail.com>:
>> >>>>
>> >>>>> segment detail page shows:
>> >>>>>
>> >>>>> Start Date (Include) 2015-03-15 00:00:00
>> >>>>> End Date (Exclude) 2015-03-16 00:00:00
>> >>>>>
>> >>>>> 2015-03-16 19:11 GMT+08:00 dong wang <el...@gmail.com>:
>> >>>>>
>> >>>>>> The latest segment is the newest one which contains today's data,
>> >>>>>>due
>> >>>>>> to some incorrect data, I want to refresh the segment, however,
>>when
>> >>>>>> cube->actions->refresh->select the latest segment, then the
>> >>>>>>following error
>> >>>>>> occurs:
>> >>>>>>
>> >>>>>> New segment range invalid, start date must be earlier than end
>>date,
>> >>>>>> 1426464000000 < 1426464000000
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>>
>>
Re: error when refreshing the latest segment
Posted by dong wang <el...@gmail.com>.
Shaofeng, I think mine is a little different from what you mentioned:
1, the partition date column can be recognized correctly
2, the segment is built successfully already
3, I just refresh the LATEST successful segment due to some base data
changes
2015-03-16 22:03 GMT+08:00 Shi, Shaofeng <sh...@ebay.com>:
> Hi Dong, which version of the Kylin build that you deployed?
>
> In last week we found a bug which will cause Kylin fail to allocate a new
> a segment; Kylin will report error saying that the startDate is not
> matched with expectation; Here is the JIRA:
>
> https://issues.apache.org/jira/browse/KYLIN-628
>
> If you¹re using Kylin¹s binary package, please update to the latest one on
> https://kylin.incubator.apache.org/download/
> If you¹re running from source code, please pull the latest code on staging
> branch;
>
> On 3/16/15, 7:40 PM, "dong wang" <el...@gmail.com> wrote:
>
> >since the following codes pass invalid startdate and enddate, which leads
> >to the issue
> >
> > public CubeSegment appendSegments(CubeInstance cube, long endDate)
> >throws
> >IOException {
> > checkNoBuildingSegment(cube);
> >
> > CubeSegment newSegment;
> > if
> >(cube.getDescriptor().getModel().getPartitionDesc().isPartitioned()) {
> > long startDate = calculateStartDateForAppendSegment(cube);
> > newSegment = newSegment(cube, startDate, endDate);
> > } else {
> > newSegment = newSegment(cube, 0, Long.MAX_VALUE);
> > }
> >
> > validateNewSegments(cube, newSegment);
> > cube.getSegments().add(newSegment);
> > Collections.sort(cube.getSegments());
> > updateCube(cube);
> >
> > return newSegment;
> > }
> >
> >2015-03-16 19:38 GMT+08:00 dong wang <el...@gmail.com>:
> >
> >> The related codes, since I'm not that clear about the logic it should
> >>be:
> >>
> >> private CubeSegment newSegment(CubeInstance cubeInstance, long
> >> startDate, long endDate) {
> >> if (startDate >= endDate)
> >> throw new IllegalArgumentException("New segment range
> >>invalid,
> >> start date must be earlier than end date, " + startDate + " < " +
> >>endDate);
> >>
> >> CubeSegment segment = new CubeSegment();
> >> String incrementalSegName =
> >>CubeSegment.getSegmentName(startDate,
> >> endDate);
> >> segment.setUuid(UUID.randomUUID().toString());
> >> segment.setName(incrementalSegName);
> >> Date creatTime = new Date();
> >> segment.setCreateTimeUTC(creatTime.getTime());
> >> segment.setDateRangeStart(startDate);
> >> segment.setDateRangeEnd(endDate);
> >> segment.setStatus(SegmentStatusEnum.NEW);
> >> segment.setStorageLocationIdentifier(generateStorageLocation());
> >>
> >> segment.setCubeInstance(cubeInstance);
> >>
> >> segment.validate();
> >> return segment;
> >> }
> >>
> >> 2015-03-16 19:29 GMT+08:00 dong wang <el...@gmail.com>:
> >>
> >>> more notes. this segment to be refreshed is built sccessful first,
> >>>then,
> >>> I plan to refresh it due to some incorrect data.
> >>>
> >>> 2015-03-16 19:25 GMT+08:00 dong wang <el...@gmail.com>:
> >>>
> >>>> watched with firebug, the parameters pass to the backend seem to be
> >>>> CORRECT as below:
> >>>>
> >>>> {buildType: "BUILD", startTime: 1426377600000, endTime: 1426464000000}
> >>>>
> >>>> 2015-03-16 19:15 GMT+08:00 dong wang <el...@gmail.com>:
> >>>>
> >>>>> segment detail page shows:
> >>>>>
> >>>>> Start Date (Include) 2015-03-15 00:00:00
> >>>>> End Date (Exclude) 2015-03-16 00:00:00
> >>>>>
> >>>>> 2015-03-16 19:11 GMT+08:00 dong wang <el...@gmail.com>:
> >>>>>
> >>>>>> The latest segment is the newest one which contains today's data,
> >>>>>>due
> >>>>>> to some incorrect data, I want to refresh the segment, however, when
> >>>>>> cube->actions->refresh->select the latest segment, then the
> >>>>>>following error
> >>>>>> occurs:
> >>>>>>
> >>>>>> New segment range invalid, start date must be earlier than end date,
> >>>>>> 1426464000000 < 1426464000000
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: error when refreshing the latest segment
Posted by "Shi, Shaofeng" <sh...@ebay.com>.
Hi Dong, which version of the Kylin build that you deployed?
In last week we found a bug which will cause Kylin fail to allocate a new
a segment; Kylin will report error saying that the startDate is not
matched with expectation; Here is the JIRA:
https://issues.apache.org/jira/browse/KYLIN-628
If you¹re using Kylin¹s binary package, please update to the latest one on
https://kylin.incubator.apache.org/download/
If you¹re running from source code, please pull the latest code on staging
branch;
On 3/16/15, 7:40 PM, "dong wang" <el...@gmail.com> wrote:
>since the following codes pass invalid startdate and enddate, which leads
>to the issue
>
> public CubeSegment appendSegments(CubeInstance cube, long endDate)
>throws
>IOException {
> checkNoBuildingSegment(cube);
>
> CubeSegment newSegment;
> if
>(cube.getDescriptor().getModel().getPartitionDesc().isPartitioned()) {
> long startDate = calculateStartDateForAppendSegment(cube);
> newSegment = newSegment(cube, startDate, endDate);
> } else {
> newSegment = newSegment(cube, 0, Long.MAX_VALUE);
> }
>
> validateNewSegments(cube, newSegment);
> cube.getSegments().add(newSegment);
> Collections.sort(cube.getSegments());
> updateCube(cube);
>
> return newSegment;
> }
>
>2015-03-16 19:38 GMT+08:00 dong wang <el...@gmail.com>:
>
>> The related codes, since I'm not that clear about the logic it should
>>be:
>>
>> private CubeSegment newSegment(CubeInstance cubeInstance, long
>> startDate, long endDate) {
>> if (startDate >= endDate)
>> throw new IllegalArgumentException("New segment range
>>invalid,
>> start date must be earlier than end date, " + startDate + " < " +
>>endDate);
>>
>> CubeSegment segment = new CubeSegment();
>> String incrementalSegName =
>>CubeSegment.getSegmentName(startDate,
>> endDate);
>> segment.setUuid(UUID.randomUUID().toString());
>> segment.setName(incrementalSegName);
>> Date creatTime = new Date();
>> segment.setCreateTimeUTC(creatTime.getTime());
>> segment.setDateRangeStart(startDate);
>> segment.setDateRangeEnd(endDate);
>> segment.setStatus(SegmentStatusEnum.NEW);
>> segment.setStorageLocationIdentifier(generateStorageLocation());
>>
>> segment.setCubeInstance(cubeInstance);
>>
>> segment.validate();
>> return segment;
>> }
>>
>> 2015-03-16 19:29 GMT+08:00 dong wang <el...@gmail.com>:
>>
>>> more notes. this segment to be refreshed is built sccessful first,
>>>then,
>>> I plan to refresh it due to some incorrect data.
>>>
>>> 2015-03-16 19:25 GMT+08:00 dong wang <el...@gmail.com>:
>>>
>>>> watched with firebug, the parameters pass to the backend seem to be
>>>> CORRECT as below:
>>>>
>>>> {buildType: "BUILD", startTime: 1426377600000, endTime: 1426464000000}
>>>>
>>>> 2015-03-16 19:15 GMT+08:00 dong wang <el...@gmail.com>:
>>>>
>>>>> segment detail page shows:
>>>>>
>>>>> Start Date (Include) 2015-03-15 00:00:00
>>>>> End Date (Exclude) 2015-03-16 00:00:00
>>>>>
>>>>> 2015-03-16 19:11 GMT+08:00 dong wang <el...@gmail.com>:
>>>>>
>>>>>> The latest segment is the newest one which contains today's data,
>>>>>>due
>>>>>> to some incorrect data, I want to refresh the segment, however, when
>>>>>> cube->actions->refresh->select the latest segment, then the
>>>>>>following error
>>>>>> occurs:
>>>>>>
>>>>>> New segment range invalid, start date must be earlier than end date,
>>>>>> 1426464000000 < 1426464000000
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
Re: error when refreshing the latest segment
Posted by dong wang <el...@gmail.com>.
since the following codes pass invalid startdate and enddate, which leads
to the issue
public CubeSegment appendSegments(CubeInstance cube, long endDate) throws
IOException {
checkNoBuildingSegment(cube);
CubeSegment newSegment;
if
(cube.getDescriptor().getModel().getPartitionDesc().isPartitioned()) {
long startDate = calculateStartDateForAppendSegment(cube);
newSegment = newSegment(cube, startDate, endDate);
} else {
newSegment = newSegment(cube, 0, Long.MAX_VALUE);
}
validateNewSegments(cube, newSegment);
cube.getSegments().add(newSegment);
Collections.sort(cube.getSegments());
updateCube(cube);
return newSegment;
}
2015-03-16 19:38 GMT+08:00 dong wang <el...@gmail.com>:
> The related codes, since I'm not that clear about the logic it should be:
>
> private CubeSegment newSegment(CubeInstance cubeInstance, long
> startDate, long endDate) {
> if (startDate >= endDate)
> throw new IllegalArgumentException("New segment range invalid,
> start date must be earlier than end date, " + startDate + " < " + endDate);
>
> CubeSegment segment = new CubeSegment();
> String incrementalSegName = CubeSegment.getSegmentName(startDate,
> endDate);
> segment.setUuid(UUID.randomUUID().toString());
> segment.setName(incrementalSegName);
> Date creatTime = new Date();
> segment.setCreateTimeUTC(creatTime.getTime());
> segment.setDateRangeStart(startDate);
> segment.setDateRangeEnd(endDate);
> segment.setStatus(SegmentStatusEnum.NEW);
> segment.setStorageLocationIdentifier(generateStorageLocation());
>
> segment.setCubeInstance(cubeInstance);
>
> segment.validate();
> return segment;
> }
>
> 2015-03-16 19:29 GMT+08:00 dong wang <el...@gmail.com>:
>
>> more notes. this segment to be refreshed is built sccessful first, then,
>> I plan to refresh it due to some incorrect data.
>>
>> 2015-03-16 19:25 GMT+08:00 dong wang <el...@gmail.com>:
>>
>>> watched with firebug, the parameters pass to the backend seem to be
>>> CORRECT as below:
>>>
>>> {buildType: "BUILD", startTime: 1426377600000, endTime: 1426464000000}
>>>
>>> 2015-03-16 19:15 GMT+08:00 dong wang <el...@gmail.com>:
>>>
>>>> segment detail page shows:
>>>>
>>>> Start Date (Include) 2015-03-15 00:00:00
>>>> End Date (Exclude) 2015-03-16 00:00:00
>>>>
>>>> 2015-03-16 19:11 GMT+08:00 dong wang <el...@gmail.com>:
>>>>
>>>>> The latest segment is the newest one which contains today's data, due
>>>>> to some incorrect data, I want to refresh the segment, however, when
>>>>> cube->actions->refresh->select the latest segment, then the following error
>>>>> occurs:
>>>>>
>>>>> New segment range invalid, start date must be earlier than end date,
>>>>> 1426464000000 < 1426464000000
>>>>>
>>>>
>>>>
>>>
>>
>
Re: error when refreshing the latest segment
Posted by dong wang <el...@gmail.com>.
The related codes, since I'm not that clear about the logic it should be:
private CubeSegment newSegment(CubeInstance cubeInstance, long startDate,
long endDate) {
if (startDate >= endDate)
throw new IllegalArgumentException("New segment range invalid,
start date must be earlier than end date, " + startDate + " < " + endDate);
CubeSegment segment = new CubeSegment();
String incrementalSegName = CubeSegment.getSegmentName(startDate,
endDate);
segment.setUuid(UUID.randomUUID().toString());
segment.setName(incrementalSegName);
Date creatTime = new Date();
segment.setCreateTimeUTC(creatTime.getTime());
segment.setDateRangeStart(startDate);
segment.setDateRangeEnd(endDate);
segment.setStatus(SegmentStatusEnum.NEW);
segment.setStorageLocationIdentifier(generateStorageLocation());
segment.setCubeInstance(cubeInstance);
segment.validate();
return segment;
}
2015-03-16 19:29 GMT+08:00 dong wang <el...@gmail.com>:
> more notes. this segment to be refreshed is built sccessful first, then,
> I plan to refresh it due to some incorrect data.
>
> 2015-03-16 19:25 GMT+08:00 dong wang <el...@gmail.com>:
>
>> watched with firebug, the parameters pass to the backend seem to be
>> CORRECT as below:
>>
>> {buildType: "BUILD", startTime: 1426377600000, endTime: 1426464000000}
>>
>> 2015-03-16 19:15 GMT+08:00 dong wang <el...@gmail.com>:
>>
>>> segment detail page shows:
>>>
>>> Start Date (Include) 2015-03-15 00:00:00
>>> End Date (Exclude) 2015-03-16 00:00:00
>>>
>>> 2015-03-16 19:11 GMT+08:00 dong wang <el...@gmail.com>:
>>>
>>>> The latest segment is the newest one which contains today's data, due
>>>> to some incorrect data, I want to refresh the segment, however, when
>>>> cube->actions->refresh->select the latest segment, then the following error
>>>> occurs:
>>>>
>>>> New segment range invalid, start date must be earlier than end date,
>>>> 1426464000000 < 1426464000000
>>>>
>>>
>>>
>>
>
Re: error when refreshing the latest segment
Posted by dong wang <el...@gmail.com>.
more notes. this segment to be refreshed is built sccessful first, then, I
plan to refresh it due to some incorrect data.
2015-03-16 19:25 GMT+08:00 dong wang <el...@gmail.com>:
> watched with firebug, the parameters pass to the backend seem to be
> CORRECT as below:
>
> {buildType: "BUILD", startTime: 1426377600000, endTime: 1426464000000}
>
> 2015-03-16 19:15 GMT+08:00 dong wang <el...@gmail.com>:
>
>> segment detail page shows:
>>
>> Start Date (Include) 2015-03-15 00:00:00
>> End Date (Exclude) 2015-03-16 00:00:00
>>
>> 2015-03-16 19:11 GMT+08:00 dong wang <el...@gmail.com>:
>>
>>> The latest segment is the newest one which contains today's data, due to
>>> some incorrect data, I want to refresh the segment, however, when
>>> cube->actions->refresh->select the latest segment, then the following error
>>> occurs:
>>>
>>> New segment range invalid, start date must be earlier than end date,
>>> 1426464000000 < 1426464000000
>>>
>>
>>
>
Re: error when refreshing the latest segment
Posted by dong wang <el...@gmail.com>.
watched with firebug, the parameters pass to the backend seem to be CORRECT
as below:
{buildType: "BUILD", startTime: 1426377600000, endTime: 1426464000000}
2015-03-16 19:15 GMT+08:00 dong wang <el...@gmail.com>:
> segment detail page shows:
>
> Start Date (Include) 2015-03-15 00:00:00
> End Date (Exclude) 2015-03-16 00:00:00
>
> 2015-03-16 19:11 GMT+08:00 dong wang <el...@gmail.com>:
>
>> The latest segment is the newest one which contains today's data, due to
>> some incorrect data, I want to refresh the segment, however, when
>> cube->actions->refresh->select the latest segment, then the following error
>> occurs:
>>
>> New segment range invalid, start date must be earlier than end date,
>> 1426464000000 < 1426464000000
>>
>
>
Re: error when refreshing the latest segment
Posted by dong wang <el...@gmail.com>.
segment detail page shows:
Start Date (Include) 2015-03-15 00:00:00
End Date (Exclude) 2015-03-16 00:00:00
2015-03-16 19:11 GMT+08:00 dong wang <el...@gmail.com>:
> The latest segment is the newest one which contains today's data, due to
> some incorrect data, I want to refresh the segment, however, when
> cube->actions->refresh->select the latest segment, then the following error
> occurs:
>
> New segment range invalid, start date must be earlier than end date,
> 1426464000000 < 1426464000000
>