[jira] [Comment Edited] (KYLIN-6087) Support built-in auto scheduled segment build with model-level config

["huangsheng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104041#comment-18104041 ] 

huangsheng edited comment on KYLIN-6087 at 9/1/26 11:34 AM:
------------------------------------------------------------

h1. Dev Design
h2. Background

Kylin supports manual incremental segment builds, but users otherwise need an external scheduler when models must build fixed logical-date ranges periodically.

This feature provides built-in automatic segment scheduling with two levels of control:
 * A project-level switch determines whether the dispatcher processes the project.
 * Each model retains its own schedule and target-range configuration.

The dispatcher only submits standard Kylin build jobs. It does not execute Spark workloads directly.
h2. Configuration
h3. Project-Level Switch

Automatic segment dispatch is disabled by default and must be enabled for each project:

 

{{{
  "auto_segment_build_enabled": true
}}}

The switch is available under {*}Project Settings > Basic Settings > Segment Settings{*}.

Disabling the project switch stops automatic dispatch without deleting or modifying model-level configurations.
h3. Model-Level Configuration

Each model stores its own {{auto_segment_build}} configuration:

 

{{{
  "enabled": true,
  "trigger_time": "01:00:00",
  "logical_date_offset_days": 1,
  "data_range_start_time": "00:00:00",
  "data_range_end_time": "24:00:00"
}}}

Field semantics:
||Field||Description||
|{{enabled}}|Enables automatic segment builds for this model.|
|{{trigger_time}}|Daily trigger time in the project timezone, formatted as {{{}HH:mm:ss{}}}.|
|{{logical_date_offset_days}}|Number of days subtracted from the scheduled trigger date. It must be at least {{{}1{}}}.|
|{{data_range_start_time}}|Start time on the calculated logical date.|
|{{data_range_end_time}}|End time on the logical date. {{24:00:00}} represents midnight at the end of that date.|

{{24:00:00}} is accepted only for {{{}data_range_end_time{}}}. Other time values must be valid {{HH:mm:ss}} values.

Automatic segment builds are not supported for:
 * Broken models
 * Streaming models
 * Full-load models without a partition column
 * Multi-partition models

Offline batch models remain eligible. Model online status does not control automatic segment scheduling.
h3. Dispatcher Interval

The dispatcher uses a server-level fixed-delay configuration:

 

{{kylin.model.auto-segment-build.dispatcher-interval-ms=30000}}

The default interval is 30 seconds. The next scan starts after the previous scan completes and the configured delay elapses.

This interval controls how frequently Kylin checks schedules. It does not replace the model-level {{{}trigger_time{}}}.
h2. Scheduling Architecture

A single global dispatcher scans metadata instead of registering one in-memory cron task for every model.

For each execution, the dispatcher:
 # Verifies that the current node is the job-scheduler master.
 # Uses an in-process atomic guard to prevent overlapping dispatcher executions.
 # Enumerates all projects.
 # Skips projects whose {{auto_segment_build_enabled}} switch is disabled.
 # Enumerates healthy underlying models, including both online and offline models.
 # Filters models that are unsupported or whose model-level schedule is disabled or invalid.
 # Calculates the latest scheduled occurrence that is not later than the current time.
 # Calculates and normalizes the target segment range.
 # Evaluates existing segments and build jobs against the target range.
 # Calls {{ModelBuildService.incrementBuildSegmentsByScheduler}} when submission is allowed.

{{ModelBuildService}} creates the segment and persists a normal {{INC_BUILD}} job using the scheduler submitter identity.
h2. Schedule Calculation

The scheduled occurrence is calculated in the project timezone:

 

{{if today's trigger time <= current time:
    scheduled occurrence = today's trigger
else:
    scheduled occurrence = yesterday's trigger}}

The logical date and target range are then calculated from that scheduled occurrence:

 

{{logical date = scheduled local date - logical_date_offset_days
target start = logical date + data_range_start_time
target end   = logical date + data_range_end_time}}

For example, with:

 

{{Project timezone: Asia/Shanghai
Current time:     2026-09-01 10:00:00
Trigger time:     01:00:00
Offset:           1 day
Data range:       00:00:00 - 24:00:00}}

The latest scheduled occurrence is {{{}2026-09-01 01:00:00{}}}, and the target range is:

 

{{[2026-08-31 00:00:00, 2026-09-01 00:00:00)}}

The calculated timestamps are normalized with the model partition-date format before being compared with existing segment ranges. This keeps dispatcher decisions consistent with {{{}ModelBuildService{}}}.
h2. Trigger Reliability

The dispatcher no longer relies on a {{(previousScanTime, currentScanTime]}} trigger window.

Every scan evaluates the latest due schedule cycle. This provides the following behavior:
 * A scan delayed beyond {{trigger_time}} still processes the cycle.
 * A process restart after {{trigger_time}} still processes the cycle.
 * A master takeover after {{trigger_time}} still processes the cycle.
 * Enabling a project or model after {{trigger_time}} processes the latest due cycle.

Only the latest due cycle is evaluated. The dispatcher does not replay every historical cycle after a multi-day outage.

Each cycle is identified by:

 

{{project / model / scheduled occurrence / normalized target range}}

An in-memory cache tracks up to 10,000 cycle states and expires entries after two days of inactivity.

Only fully covered cycles are treated as complete and skipped without further metadata checks. Other states are re-evaluated during subsequent scans.

The cache is an optimization rather than the source of correctness. After restart, failover, or cache eviction, existing segment metadata is used to determine whether the cycle has already been completed.
h2. Range-Aware Conflict Handling

The scheduler compares the normalized target range with existing {{{}READY{}}}, {{{}WARNING{}}}, and {{NEW}} segments.
||State||Condition||Action||
|{{COVERED}}|One segment, a superset segment, or a contiguous union of {{READY}} segments completely covers the target.|Mark the cycle complete and do not submit.|
|{{COVERED_WITH_WARNING}}|The target is completely covered, but required coverage includes a {{WARNING}} segment.|Mark the cycle complete, do not submit, and log a warning.|
|{{IN_PROGRESS}}|An overlapping {{NEW}} segment is associated with a {{{}READY{}}}, {{{}PENDING{}}}, or {{RUNNING}} build job.|Wait and re-evaluate on the next dispatcher scan.|
|{{BLOCKED}}|An overlapping {{NEW}} segment is associated with a non-progressing job, such as {{ERROR}} or {{{}PAUSED{}}}, or has no corresponding job.|Do not submit and continue re-evaluating.|
|{{PARTIAL_OVERLAP}}|Existing {{READY}} or {{WARNING}} segments overlap the target but do not completely cover it.|Do not automatically replace or expand the existing segments. Continue re-evaluating.|
|{{NO_OVERLAP}}|No existing segment overlaps the target.|Submit the automatic build.|
|{{SUBMITTED}}|{{ModelBuildService}} accepted the request.|Re-evaluate metadata during the next scan until coverage is complete.|
|{{RETRYABLE_FAILURE}}|Submission failed with an exception.|Retry during the next dispatcher scan.|

Completed coverage is evaluated before in-progress segments. Therefore, if usable segments already cover the complete target, no additional automatic build is submitted.
h2. Boundary Behavior

The following cases are handled explicitly:
||Existing condition||Result||
|The exact target range was manually built successfully.|Skip because the target is covered.|
|A larger successful segment contains the target range.|Skip because the target is covered.|
|Multiple adjacent successful segments collectively cover the target.|Skip because the target is covered.|
|An exact-range segment is currently building.|Wait and re-evaluate.|
|A larger segment containing the target is currently building.|Wait; after success, the target becomes covered.|
|A smaller segment inside the target is currently building.|Wait; after success, the result becomes a partial-overlap conflict.|
|A segment partially intersects the target and is currently building.|Wait; after success, the result becomes a partial-overlap conflict.|
|Another segment build is running on a disjoint range.|Allow the scheduled target to be submitted.|
|An overlapping build job is paused or failed.|Block submission until metadata or job state is resolved.|
|An overlapping {{NEW}} segment has no corresponding job.|Treat it as an orphan and block submission.|
|Submission fails temporarily.|Retry during the next dispatcher scan.|

The scheduler intentionally does not delete existing segments, split the target range, build only missing holes, or replace manually created partial ranges.
h2. Concurrency Control

Concurrency control is range-aware rather than model-wide.

A progressing build job blocks automatic submission only when its target segment overlaps the scheduled target range. Build jobs on disjoint ranges do not block submission.

Correctness is protected at multiple levels:
 * Only the job-scheduler master runs the dispatcher.
 * The in-process atomic guard serializes dispatcher scans.
 * Existing segment metadata provides idempotency.
 * Kylin job submission validates overlapping running jobs.
 * Segment creation and job persistence run inside existing metadata transactions.
 * Failed submissions remain retryable.

These existing transactional and overlap checks remain the final consistency guard during master transitions or concurrent manual submissions.
h2. Observability

The dispatcher logs state transitions with the project, model, target range, and related segments.

Repeated unchanged states do not repeatedly emit the same warning. Repeated submission failures are downgraded to debug logging after the initial error.

Invalid model configurations are skipped, and identical validation warnings are suppressed for one day.
h2. Compatibility

The existing model-level {{auto_segment_build}} metadata, REST API, and UI configuration remain unchanged.

The updated implementation adds:
 * A project-level {{auto_segment_build_enabled}} switch
 * A fixed-delay dispatcher interval
 * Latest-cycle catch-up behavior
 * Range-aware conflict detection
 * Retry of unresolved or transient states

Existing projects default to project-level automatic dispatch being disabled. Existing model configurations remain stored and become active when the project switch is enabled.

No new Spark execution path or job type is introduced. Automatic builds continue to use the standard {{INC_BUILD}} workflow.


was (Author: JIRAUSER297438):
h3. Dev Design
h3. 
Background

Kylin supports manual incremental segment builds, but users need an external scheduler when a model must build a fixed logical-day range periodically. This feature adds built-in automatic segment build scheduling while preserving model-level configuration and workload isolation.
h3. Configuration

Each model stores its own {{auto_segment_build}} configuration:
{code:java}
{
  "enabled": true,
  "trigger_time": "01:00:00",
  "logical_date_offset_days": 1,
  "data_range_start_time": "00:00:00",
  "data_range_end_time": "24:00:00"
} {code}
 

 

The dispatcher frequency is a server-level configuration:
{code:java}
kylin.model.auto-segment-build.dispatcher-cron=*/30 * * * * ? {code}
The dispatcher cron controls how often Kylin checks model schedules. It does not replace the model-level {{{}trigger_time{}}}.
h3. Scheduling Architecture

A single dispatcher periodically scans projects instead of registering one in-memory cron task for every model.

For each project, the dispatcher:
 # Runs only on the job-scheduler master.
 # Enumerates online and healthy models.
 # Filters out disabled, broken, offline, streaming, full-load, and multi-partition models.
 # Evaluates each model’s {{trigger_time}} using the project timezone.
 # Submits the model build only when its trigger falls in {{{}(previousScanTime, currentScanTime]{}}}.
 # Calls {{ModelBuildService}} to create the segment and persist a normal {{INC_BUILD}} job.

The dispatcher does not execute Spark build work directly.
h3. Trigger Reliability

Dispatcher executions are serialized inside one process. The previous scan timestamp creates an exclusive/inclusive trigger window, preventing duplicate submission when the dispatcher runs multiple times around the same trigger.

On startup, clock rollback, or master takeover, the dispatcher uses a one-minute lookback. This avoids replaying historical builds while still covering normal scheduling jitter.

The segment range is calculated from the scheduled trigger date in the project timezone, rather than from the server’s current local date. This keeps delayed execution and timezone behavior deterministic.
h3. Concurrency Control

Before submission, the scheduler checks all progressing build-category jobs for the same model.

If any model build job is progressing, the scheduled build is skipped. This is intentionally model-level rather than segment-level because concurrent build jobs may mutate shared model and segment metadata.

The current design does not retry that daily trigger after the progressing job finishes. Existing transaction and segment-overlap validation remain the final consistency guards against concurrent master transitions or duplicate submissions.
h3. Compatibility

The existing model-level REST API, metadata structure, and UI configuration remain unchanged. The change only replaces the internal per-model {{ScheduledFuture}} implementation with a dispatcher.
h3.  

> Support built-in auto scheduled segment build with model-level config
> ---------------------------------------------------------------------
>
>                 Key: KYLIN-6087
>                 URL: https://issues.apache.org/jira/browse/KYLIN-6087
>             Project: Kylin
>          Issue Type: New Feature
>          Components: Job Engine
>    Affects Versions: 5.0.3
>            Reporter: huangsheng
>            Assignee: huangsheng
>            Priority: Minor
>
> Currently, in kylin, incremental segment build job can only be triggered manually. I would like to add an automatic timed trigger for segment build job function because now we all rely on external timing tools (such as crontab) to call the kylin incremental build api at regular intervals to start segment construction. For example, the build job of yesterday's segment is triggered at 1 a.m. every day
> I want to directly incorporate this function into kylin. My idea is to add a new entry for scheduled scheduling in the model configuration pop-up box. It is necessary to configure the daily trigger time and the time range for building this segment. There are two parameters to configure for the time range. One is the logical date, such as yesterday
> The day before yesterday, etc., one is a specific time, for example, from 0:00 to 24:00. Then the back end schedules build tasks at regular intervals based on this configuration.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)