[GitHub] [incubator-tvm] u99127 commented on issue #4304: [RFC] Benchmark Performance Log Format WIP

[GitBox <gi...@apache.org>]

u99127 commented on issue #4304: [RFC] Benchmark Performance Log Format WIP
URL: https://github.com/apache/incubator-tvm/issues/4304#issuecomment-557730728
 
 
   > 
   > 
   > I think docker_tag under metadata as long as it is optional (the RFC said the field was required which confuses me).
   > 
   > There are certainly cases when people want to limit the number of threads, e.g. use only the little cores on the phone to save power. While these could have been a nested field in the hardware_config, it makes the field quite complicated.
   
   
   
   > 
   > One of the key property of we should push for the log format is its simplicity. The three fields(engine, workload, hardware_config) are three fields that the user will mostly visit. e.g. ask how can "tvm" do for "resnet-18" on "rasp3b"
   
   Specifically Rasp3b is fine because it's not big Little . It only has Cortex-A53 cores. 
   
   If the run of the benchmarking is able to pin tasks on specific cores for b.L systems, then I don't see how you can avoid that - unless the TVM stack can only run on the big or only on the little. 
   
   > 
   > Of course the information was incomplete, but the in many cases the three fields already gives a quite complete picture, assuming the default implied target and maximum usage of the resources.
   > 
   
   
   Sorry, that  puzzles me again - missing hardware configuration will cause confusion, it certainly  misses some information. 
   
   Not capturing that information would result in user confusion and comparison of apples and oranges. I appreciate the push towards simplicity but it feels that it's at the cost of accuracy. If I were looking at it and I wanted to know the difference then I would prefer to look at it. It's quite likely that if we were looking for performance monitoring over time, then figuring out a way of distinguishing between pinning tasks on big vs pinning tasks on little vs using EAS across the board and therefore note pinning would need to be capture.
   
   > Given that number of threads and other runtime configurations(thread pool setups) are indeed runtime properties that was different from software_config. I think we should keep the original runtime_config field for these information.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services