You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "ffacs (via GitHub)" <gi...@apache.org> on 2024/04/15 15:19:11 UTC
[PR] ORC-1688: [C++] Do not access TZDB if there is no timestamp type [orc]
ffacs opened a new pull request, #1893:
URL: https://github.com/apache/orc/pull/1893
### What changes were proposed in this pull request?
1. Don't write writer's Time Zone name when there are no timestamp types.
2. Do not access TZDB if there is no timestamp type.
### Why are the changes needed?
This patch could accelerate reading by reducing unnecessary IO.
### How was this patch tested?
All UT passed.
### Was this patch authored or co-authored using generative AI tooling?
NO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1688: [C++] Do not access TZDB if there is no timestamp type [orc]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac closed pull request #1893: ORC-1688: [C++] Do not access TZDB if there is no timestamp type
URL: https://github.com/apache/orc/pull/1893
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1688: [C++] Do not access TZDB if there is no timestamp type [orc]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #1893:
URL: https://github.com/apache/orc/pull/1893#discussion_r1566617237
##########
c++/test/TestType.cc:
##########
@@ -490,4 +490,28 @@ namespace orc {
*(footer.add_types()) = intType;
checkProtoTypes(footer);
}
+
+ TEST(TestType, testExistSubType) {
Review Comment:
Could you also add a roundtrip test to write and read ORC file without timestamp type with an invalid TZDIR env set? Something like we did in `TEST(TestTimezone, testMissingTZDB)` and be careful about the timezone cache.
##########
c++/src/Reader.cc:
##########
@@ -1089,14 +1089,9 @@ namespace orc {
} while (sargsApplier_ && currentStripe_ < lastStripe_);
if (currentStripe_ < lastStripe_) {
- // get writer timezone info from stripe footer to help understand timestamp values.
- const Timezone& writerTimezone =
- currentStripeFooter_.has_writer_timezone()
- ? getTimezoneByName(currentStripeFooter_.writer_timezone())
- : localTimezone_;
StripeStreamsImpl stripeStreams(*this, currentStripe_, currentStripeInfo_,
currentStripeFooter_, currentStripeInfo_.offset(),
- *contents_->stream, writerTimezone, readerTimezone_);
+ *contents_->stream, readerTimezone_);
Review Comment:
We should also fix here: https://github.com/apache/orc/blob/main/c%2B%2B/src/Reader.cc#L249-L257
It blindly gets timezone when creating a RowReader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1688: [C++] Do not access TZDB if there is no timestamp type [orc]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1893:
URL: https://github.com/apache/orc/pull/1893#issuecomment-2061597913
Nice fix! Will take a deeper look at it tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1688: [C++] Do not access TZDB if there is no timestamp type [orc]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #1893:
URL: https://github.com/apache/orc/pull/1893#discussion_r1574667774
##########
c++/src/Timezone.cc:
##########
@@ -691,12 +683,68 @@ namespace orc {
try {
std::unique_ptr<InputStream> file = readFile(filename);
size_t size = static_cast<size_t>(file->getLength());
- std::vector<unsigned char> buffer(size);
+ buffer.resize(size);
file->read(&buffer[0], size, 0);
- timezoneCache[filename] = std::make_shared<TimezoneImpl>(filename, buffer);
} catch (ParseError& err) {
throw TimezoneError(err.what());
}
+ return buffer;
+ }
+
+ class LazyTimezone : public Timezone {
+ private:
+ std::string filename_;
+ mutable std::unique_ptr<TimezoneImpl> impl_;
+ mutable std::atomic<bool> initialized_;
+
+ TimezoneImpl* getImpl() const {
+ if (!initialized_) {
+ std::lock_guard<std::mutex> timezone_lock(timezone_mutex);
Review Comment:
`timezone_mutex` is used to guard `TimezoneImpl::timezoneCache`. Should we use a much finer-grained mutex which is per LazyTimezone instance?
##########
c++/src/Timezone.cc:
##########
@@ -691,12 +683,68 @@ namespace orc {
try {
std::unique_ptr<InputStream> file = readFile(filename);
size_t size = static_cast<size_t>(file->getLength());
- std::vector<unsigned char> buffer(size);
+ buffer.resize(size);
file->read(&buffer[0], size, 0);
- timezoneCache[filename] = std::make_shared<TimezoneImpl>(filename, buffer);
} catch (ParseError& err) {
throw TimezoneError(err.what());
}
+ return buffer;
+ }
+
+ class LazyTimezone : public Timezone {
+ private:
+ std::string filename_;
+ mutable std::unique_ptr<TimezoneImpl> impl_;
+ mutable std::atomic<bool> initialized_;
+
+ TimezoneImpl* getImpl() const {
+ if (!initialized_) {
+ std::lock_guard<std::mutex> timezone_lock(timezone_mutex);
+ if (!initialized_) {
+ auto buffer = loadTZDB(filename_);
+ impl_.reset(new TimezoneImpl(filename_, std::move(buffer)));
Review Comment:
```suggestion
impl_ = std::make_unique<TimezoneImpl>(filename_, std::move(buffer));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1688: [C++] Do not access TZDB if there is no timestamp type [orc]
Posted by "ffacs (via GitHub)" <gi...@apache.org>.
ffacs commented on PR #1893:
URL: https://github.com/apache/orc/pull/1893#issuecomment-2061038399
This implementation should be better than prior one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org