[Apache TVM Discuss] [Questions] [auto-scheduler] Measure time for default schedule of single workload

[Wheest via Apache TVM Discuss <no...@discuss.tvm.ai>]

I am developing in Ansor, which tunes and evaluates workloads in a model independently, and can evaluate them standalone using the utilities in [measure.py](https://github.com/apache/tvm/blob/main/python/tvm/auto_scheduler/measure.py).

I want to compile a single workload in a model using the default schedule, to compare against the time for an auto-scheduled approach.

E.g.:

* Workload 1 untuned time: 1 ms
* Workload 1 tuned time: 0.5 ms
* Workload 2 untuned time: 1.5ms
* Workload 2 tuned time: 1ms
* ...
* Full model untuned inference time: 50ms
* Full model tuned inference time: 35ms

Given a tuned log file for a model, I can easily measure the inference time of a single workload with this function (full code linked below):

```python3
def test_single_workload(workload_num: int, logfile: os.PathLike) -> float:
    """Measures the time for a single workload with a given autoschedule file in Ansor

    :param workload_num: the workload number to evaluate
    :param logfile: the path to the logfile containing workload and schedule information
    :returns: Inference time of workload

    """
    reader = RecordReader(logfile)
    inputs, _ = reader.read_lines()

    timeout, n_parallel = 3, 1
    runs = 5

    inputs = [inputs[workload_num] for _ in range(runs)]
    results = local_builder_build(inputs, timeout, n_parallel)
    total = 0
    vs = []
    for r in results:
        if r.error_no == 0:
            # acceptable exit code
            vs.append(r.time_cost)
        else:
            print("Unexpected exit code:", r.error_no)
            exit(1)

    cost = np.median(vs)
    return cost
```

However, if I want to get the performance of that workload with the default schedule (no logfile), then I am unsure what I need to do.

If I wanted an equivalent function with the untuned task, I can create an input like this:

```python3
input = MeasureInput(task, state)
```

However I need a [State object](https://github.com/apache/tvm/blob/main/python/tvm/auto_scheduler/loop_state.py#L71), which describes the schedule we are evaluating, and I do not have a logfile which contains a description of the default schedule for that workload.  It cannot be set to `None`, at least the way I have tried it.

How would I get a State object for this workload in its untuned state?

A simple standalone example of my code is available at [this gist](https://gist.github.com/Wheest/28e1c1bc6135230961189644aeff8070).





---
[Visit Topic](https://discuss.tvm.apache.org/t/auto-scheduler-measure-time-for-default-schedule-of-single-workload/10730/1) to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/31cfc9e8e952352a8934ef340e4fa3ca6667323938f2a80576f77fb091488f31).

[Apache TVM Discuss] [Questions] [auto-scheduler] Measure time for default schedule of single workload

[Andrey Malyshev via Apache TVM Discuss <no...@discuss.tvm.ai>]

Autoscheduler completely ignores manually developed schedules, it creates new schedules using only compute function and its Ansor rulls how to transforms one schedule to another. You will not be able to get default scheduling through Ansor workflow. Instead of this it might be better to take into AutoTVM flow, probably. It works exactly with handy made schedules.





---
[Visit Topic](https://discuss.tvm.apache.org/t/auto-scheduler-measure-time-for-default-schedule-of-single-workload/10730/2) to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/2bc0aff044f103650b51971f33043fd45ecfab05711fa449d98579eaaf0f99b7).