You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/05/04 14:12:01 UTC
[GitHub] [incubator-tvm] siju-samuel opened a new pull request #5507: [CRT]fix to reduce RAM size during loading model
siju-samuel opened a new pull request #5507:
URL: https://github.com/apache/incubator-tvm/pull/5507
This fix will help to reduce RAM size during setup storage. I was facing issue with arduino which has only 256kb ram. For my model Setupstorage was consuming 80kb of ram and with this fix i could reduce it to 40kb. In the current code always float32 datatype is used and 4bytes is used to allocate storage pool. If runnng 8/16bit models also it will allocate 32bits.
@liangfu @tmoreau89 Please help to review this code.
Note: This fix can be applied to tvm runtime as well, but not very significant there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] siju-samuel commented on pull request #5507: [CRT]fix to reduce RAM size during loading model
Posted by GitBox <gi...@apache.org>.
siju-samuel commented on pull request #5507:
URL: https://github.com/apache/incubator-tvm/pull/5507#issuecomment-625537719
@tqchen Ideally yes. There may be some calcuation issue. I will verify once again in case of crt what is happening and get back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] siju-samuel commented on pull request #5507: [CRT]fix to reduce RAM size during loading model
Posted by GitBox <gi...@apache.org>.
siju-samuel commented on pull request #5507:
URL: https://github.com/apache/incubator-tvm/pull/5507#issuecomment-625525365
@tqchen I work in a device with very limited ram availability. Actually even a few kbs is crucial for me.
Im using a quantized model and the pool is 80% unit8 data. So i dont want to allocate all pool with float and 32bit. If i consider the bit i can restrict all pool to allocate 32bit.
So while allocating space, to compute `size` we either need `bits` or `dytpe`. Now both are not available in `TVMGraphRuntimePoolEntry`. So i added bits in `TVMGraphRuntimePoolEntry`
As you suggested, we can compute `size` while finding maximum space needed and save to `pool_entry[sid].size`
But it requires either
1. while making TVMNDArray_Empty, always use `dtype.code = kDLInt` & `dtype.bits = 8`.
2. while making TVMNDArray_Empty, use Float & 32bit (like before this change) and reduce the `size` as per `bits` and align to 4bytes.
I would like to go with 2nd approach. I havnet done the modification yet. I will do the modifications, test and will update this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] siju-samuel edited a comment on pull request #5507: [CRT]fix to reduce RAM size during loading model
Posted by GitBox <gi...@apache.org>.
siju-samuel edited a comment on pull request #5507:
URL: https://github.com/apache/incubator-tvm/pull/5507#issuecomment-625525365
@tqchen. Thanks for the review.
I work in a device with very limited ram availability. Actually even a few kbs is crucial for me.
Im using a quantized model and the pool is 80% unit8 data. So i dont want to allocate all pool with float and 32bit. If i consider the bit i can restrict all pool to allocate 32bit.
So while allocating space, to compute `size` we either need `bits` or `dytpe`. Now both are not available in `TVMGraphRuntimePoolEntry`. So i added bits in `TVMGraphRuntimePoolEntry`
As you suggested, we can compute `size` while finding maximum space needed and save to `pool_entry[sid].size`
But it requires either
1. while making TVMNDArray_Empty, always use `dtype.code = kDLInt` & `dtype.bits = 8`.
2. while making TVMNDArray_Empty, use Float & 32bit (like before this change) and reduce the `size` as per `bits` and align to 4bytes.
I would like to go with 2nd approach. I havnet done the modification yet. I will do the modifications, test and will update this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] siju-samuel edited a comment on pull request #5507: [CRT]fix to reduce RAM size during loading model
Posted by GitBox <gi...@apache.org>.
siju-samuel edited a comment on pull request #5507:
URL: https://github.com/apache/incubator-tvm/pull/5507#issuecomment-625525365
@tqchen. Thanks for the review.
I work in a device with very limited ram availability. Actually even a few kbs is crucial for me.
Im using a quantized model and the pool is 80% unit8 data. So i dont want to allocate all pool with float and 32bit. If i consider the bit i can restrict all pool to allocate 32bit.
So while allocating space, to compute `size` we either need `bits` or `dytpe`. Now both are not available in `TVMGraphRuntimePoolEntry`. So i added bits in `TVMGraphRuntimePoolEntry`
As you suggested, we can compute `size` while finding maximum space needed and save to `pool_entry[sid].size` and remove the `bits`.
But it requires either
1. while making TVMNDArray_Empty, always use `dtype.code = kDLInt` & `dtype.bits = 8`.
2. while making TVMNDArray_Empty, use Float & 32bit (like before this change) and reduce the `size` as per `bits` and align to 4bytes.
I would like to go with 2nd approach. I havnet done the modification yet. I will do the modifications, test and will update this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on pull request #5507: [CRT]fix to reduce RAM size during loading model
Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #5507:
URL: https://github.com/apache/incubator-tvm/pull/5507#issuecomment-625459772
@siju-samuel do you mind elaborate? I still do not see why do we need to specify bits here. Is it because your pool contains a lot of single element data? Since it is totally fine to allocate array using float32 and recast to int8, as long as these arrays have at least 4 elements.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] siju-samuel commented on pull request #5507: [CRT]fix to reduce RAM size during loading model
Posted by GitBox <gi...@apache.org>.
siju-samuel commented on pull request #5507:
URL: https://github.com/apache/incubator-tvm/pull/5507#issuecomment-625811064
@tqchen Thanks a lot. The 40kb ram reduced for me was due to the free of graph_json I freed immediately after saving to runtime.
The other fix on setupstorage could save me only around 200bytes totally, which is not very significant. Sorry that I overlooked the other change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on pull request #5507: [CRT]fix to reduce RAM size during loading model
Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #5507:
URL: https://github.com/apache/incubator-tvm/pull/5507#issuecomment-625527871
shouldn't the `size` argument in the pool already is reflected number of bytes(thus taking the data type into consideration?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on pull request #5507: [CRT]fix to reduce RAM size during loading model
Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #5507:
URL: https://github.com/apache/incubator-tvm/pull/5507#issuecomment-624416758
I am a bit confused by why the bits field is necessary, would it suffice to just check the dtype of each tensor and compute the maximum size correctly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] siju-samuel commented on pull request #5507: [CRT]fix to reduce RAM size during loading model
Posted by GitBox <gi...@apache.org>.
siju-samuel commented on pull request #5507:
URL: https://github.com/apache/incubator-tvm/pull/5507#issuecomment-624415991
@tqchen could you please review. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org