You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "gruuya (via GitHub)" <gi...@apache.org> on 2023/03/01 15:05:49 UTC
[GitHub] [arrow-rs] gruuya opened a new issue, #3782: Make `InMemory` object store track last modified time for each entry
gruuya opened a new issue, #3782:
URL: https://github.com/apache/arrow-rs/issues/3782
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The `InMemory` object stores currently [spoofs](https://github.com/apache/arrow-rs/blob/master/object_store/src/memory.rs#L148) the `last_modified` field of the fetched `ObjectMeta`s. In some cases, it is important for this value to be correct.
One such case is [deducing the `DeltaTable` version](https://github.com/delta-io/delta-rs/blob/main/rust/src/delta.rs#L737-L739) based on a user-provided timestamp in time travel querying—if the `last_modified` field is wrong the wrong version of the table will be loaded. Admittedly, no one should be using an in-memory store for actual work, but it would still be very handy to have this working for testing.
**Describe the solution you'd like**
Keep a separate map of `last_modified` timestamps, and update it on each write call (`put`, `copy`, etc.).
**Describe alternatives you've considered**
Don't use `InMem` store whenever `last_modified` field is crucial for the logic.
**Additional context**
This could be a good first issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #3782: Make `InMemory` object store track last modified time for each entry
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #3782:
URL: https://github.com/apache/arrow-rs/issues/3782#issuecomment-1450519872
Changing `storage` to be
```
storage: Arc<RwLock<BTreeMap<Path, (Bytes, DateTime<Utc>)>>>
```
Makes a lot of sense to me :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold closed issue #3782: Make `InMemory` object store track last modified time for each entry
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #3782: Make `InMemory` object store track last modified time for each entry
URL: https://github.com/apache/arrow-rs/issues/3782
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] gruuya commented on issue #3782: Make `InMemory` object store track last modified time for each entry
Posted by "gruuya (via GitHub)" <gi...@apache.org>.
gruuya commented on issue #3782:
URL: https://github.com/apache/arrow-rs/issues/3782#issuecomment-1451334261
> Changing `storage` to be
>
> ```
> storage: Arc<RwLock<BTreeMap<Path, (Bytes, DateTime<Utc>)>>>
> ```
>
> Makes a lot of sense to me 👍
Agreed, that is indeed more concise!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #3782: Make `InMemory` object store track last modified time for each entry
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #3782:
URL: https://github.com/apache/arrow-rs/issues/3782#issuecomment-1463940159
`label_issue.py` automatically added labels {'object-store'} from #3796
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org