You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@orc.apache.org by grundprinzip <gi...@git.apache.org> on 2017/07/25 20:50:56 UTC
[GitHub] orc pull request #142: [ORC-218] Cache timezone information in the library.
GitHub user grundprinzip opened a pull request:
https://github.com/apache/orc/pull/142
[ORC-218] Cache timezone information in the library.
Right now, for every lookup of a timezone, the library will go to
disk and parse the timezone file. While the results are cached,
doing these system calls should be avoided in environments with
restricted system calls.
This patch loads all the time zone files into the binary at
build time and instead of accessing disk for the TZ
information will simply load the buffer from memory. In addition,
this allows the user to compile the library with a static
timezone to be assumed for local use. This comes in handy to avoid
yet another set of system calls to assume the local timezone.
The following CMake flags control this feature:
* EMBEDED_TZ_DB - disables this feature if set to OFF
* STATIC_TZ=Val - sets a specific timezone
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/grundprinzip/orc orc218-tz
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/orc/pull/142.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #142
----
commit b87586d3b3b8892c15d04fe6019152deebbb8954
Author: Martin Grund <ma...@amazon.com>
Date: 2017-07-25T20:34:27Z
[ORC-218] Cache timezone information in the library.
Right now, for every lookup of a timezone, the library will go to
disk and parse the timezone file. While the results are cached,
doing these system calls should be avoided in environments with
restricted system calls.
This patch loads all the time zone files into the binary at
build time and instead of accessing disk for the TZ
information will simply load the buffer from memory. In addition,
this allows the user to compile the library with a static
timezone to be assumed for local use. This comes in handy to avoid
yet another set of system calls to assume the local timezone.
The following CMake flags control this feature:
* EMBEDED_TZ_DB - disables this feature if set to OFF
* STATIC_TZ=Val - sets a specific timezone
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc issue #142: [ORC-218] Cache timezone information in the library.
Posted by grundprinzip <gi...@git.apache.org>.
Github user grundprinzip commented on the issue:
https://github.com/apache/orc/pull/142
There's some funky stuff that's going on with the way how the code get's compiled in Travis, that I need to reproduce. The unit tests with the same combination of parameters works for me locally. Still working on it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #142: [ORC-218] Cache timezone information in the library.
Posted by grundprinzip <gi...@git.apache.org>.
Github user grundprinzip commented on a diff in the pull request:
https://github.com/apache/orc/pull/142#discussion_r131188652
--- Diff: c++/src/CMakeLists.txt ---
@@ -125,6 +125,83 @@ include_directories (
${LZ4_INCLUDE_DIRS}
)
+# To avoid reading the Timezone database from disk, we load the file during the
+# build and inject them as binary into the library. This can increase the size
+# of library but avoids doing system calls at runtime. The behavior can be
+# disabled by using the CMake flag NO_EMBEDDED_TZ_DB
+if (NOT DEFINED EMBEDDED_TZ_DB)
+ set(EMBEDDED_TZ_DB ON)
+endif()
+if (EMBEDDED_TZ_DB)
+ message(STATUS "Caching of timezone data enabled.")
+ set(DEFAULT_TIMEZONE_DIR "/usr/share/zoneinfo")
+
+ # Check if zone1970.tab is present, otherwise select zone.tab
+ # Parse zone tab to extract all availble zones
+ execute_process(COMMAND find ${DEFAULT_TIMEZONE_DIR} -type f
+ COMMAND grep -v "posix"
+ COMMAND grep -v "right"
+ COMMAND grep -v "tab"
+ RESULT_VARIABLE TZ_LIST_RC
+ OUTPUT_VARIABLE TZ_LIST_OUTPUT
+ ERROR_VARIABLE TZ_ERROR)
+
+ if (TZ_LIST_RC)
+ message(FATAL_ERROR "Could not parse Timezone database.")
+ endif()
+
+ set(FILE2ARRAY "${CMAKE_CURRENT_SOURCE_DIR}/../file2array.sh")
+
+ # For each zone generate a target
+ # Add each target to dependency on Timezone.cc
+ set(TZ_DEPS "")
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #142: [ORC-218] Cache timezone information in the library.
Posted by xndai <gi...@git.apache.org>.
Github user xndai commented on a diff in the pull request:
https://github.com/apache/orc/pull/142#discussion_r130473949
--- Diff: c++/src/CMakeLists.txt ---
@@ -125,6 +125,83 @@ include_directories (
${LZ4_INCLUDE_DIRS}
)
+# To avoid reading the Timezone database from disk, we load the file during the
+# build and inject them as binary into the library. This can increase the size
+# of library but avoids doing system calls at runtime. The behavior can be
+# disabled by using the CMake flag NO_EMBEDDED_TZ_DB
+if (NOT DEFINED EMBEDDED_TZ_DB)
+ set(EMBEDDED_TZ_DB ON)
--- End diff --
A side effect of this is when the time zone info files are updated, the Orc lib has to be recompiled and redistributed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #142: [ORC-218] Cache timezone information in the library.
Posted by xndai <gi...@git.apache.org>.
Github user xndai commented on a diff in the pull request:
https://github.com/apache/orc/pull/142#discussion_r130475038
--- Diff: c++/src/Timezone.cc ---
@@ -735,18 +739,36 @@ namespace orc {
* Get the local timezone.
*/
const Timezone& getLocalTimezone() {
+#ifdef TZ_DB_STATIC_TZ
+#define TZ_STR(x) #x
+ return getTimezoneByName(TZ_STR(TZ_DB_STATIC_TZ));
+#else
return getTimezoneByFilename(LOCAL_TIMEZONE);
+#endif
}
/**
* Get a timezone by name (eg. America/Los_Angeles).
* Results are cached.
*/
const Timezone& getTimezoneByName(const std::string& zone) {
+#ifdef USE_TZ_DB
+ {
+ std::lock_guard<std::mutex> timezone_lock(timezone_mutex);
+ std::map<std::string, std::shared_ptr<Timezone> >::iterator itr =
+ timezoneCache.find(zone);
+ if (itr != timezoneCache.end()) {
+ return *(itr->second).get();
+ }
+ timezoneCache[zone] = std::shared_ptr<Timezone>(new TimezoneImpl(zone, TZ_DATABASE[zone]));
--- End diff --
why don't we pre populate the whole time zone map so you don't need to acquire a mutex every time accessing the map?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #142: [ORC-218] Cache timezone information in the library.
Posted by majetideepak <gi...@git.apache.org>.
Github user majetideepak commented on a diff in the pull request:
https://github.com/apache/orc/pull/142#discussion_r129884718
--- Diff: c++/file2array.sh ---
@@ -0,0 +1,81 @@
+#!/bin/bash
+# Copyright 2017 Amazon.com and it's affiliates
+# Copyright 2015 Cloudera Inc.
+# Copyright 2013 University of Chicago and Argonne National Laboratory
--- End diff --
I am not sure if this Copyright information and the Apache License can co-exist.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #142: [ORC-218] Cache timezone information in the library.
Posted by grundprinzip <gi...@git.apache.org>.
Github user grundprinzip commented on a diff in the pull request:
https://github.com/apache/orc/pull/142#discussion_r131192257
--- Diff: c++/src/Timezone.cc ---
@@ -735,18 +739,36 @@ namespace orc {
* Get the local timezone.
*/
const Timezone& getLocalTimezone() {
+#ifdef TZ_DB_STATIC_TZ
+#define TZ_STR(x) #x
+ return getTimezoneByName(TZ_STR(TZ_DB_STATIC_TZ));
+#else
return getTimezoneByFilename(LOCAL_TIMEZONE);
+#endif
}
/**
* Get a timezone by name (eg. America/Los_Angeles).
* Results are cached.
*/
const Timezone& getTimezoneByName(const std::string& zone) {
+#ifdef USE_TZ_DB
+ {
+ std::lock_guard<std::mutex> timezone_lock(timezone_mutex);
+ std::map<std::string, std::shared_ptr<Timezone> >::iterator itr =
+ timezoneCache.find(zone);
+ if (itr != timezoneCache.end()) {
+ return *(itr->second).get();
+ }
+ timezoneCache[zone] = std::shared_ptr<Timezone>(new TimezoneImpl(zone, TZ_DATABASE[zone]));
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc issue #142: [ORC-218] Cache timezone information in the library.
Posted by omalley <gi...@git.apache.org>.
Github user omalley commented on the issue:
https://github.com/apache/orc/pull/142
The build is failing when the cache is enabled. I'd guess the clean up is causing a segfault.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc issue #142: [ORC-218] Cache timezone information in the library.
Posted by omalley <gi...@git.apache.org>.
Github user omalley commented on the issue:
https://github.com/apache/orc/pull/142
For most environments, this is a bad idea. Those zone files are patched all of the time. Embedding them into a library will create problems. However, it does provide a way out for environments that either have very old zone files or no copies at all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #142: [ORC-218] Cache timezone information in the library.
Posted by majetideepak <gi...@git.apache.org>.
Github user majetideepak commented on a diff in the pull request:
https://github.com/apache/orc/pull/142#discussion_r129885969
--- Diff: c++/src/CMakeLists.txt ---
@@ -125,6 +125,83 @@ include_directories (
${LZ4_INCLUDE_DIRS}
)
+# To avoid reading the Timezone database from disk, we load the file during the
+# build and inject them as binary into the library. This can increase the size
+# of library but avoids doing system calls at runtime. The behavior can be
+# disabled by using the CMake flag NO_EMBEDDED_TZ_DB
+if (NOT DEFINED EMBEDDED_TZ_DB)
+ set(EMBEDDED_TZ_DB ON)
--- End diff --
This feature should be OFF by default.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc issue #142: [ORC-218] Cache timezone information in the library.
Posted by majetideepak <gi...@git.apache.org>.
Github user majetideepak commented on the issue:
https://github.com/apache/orc/pull/142
Please add another travis-ci test with EMBEDDED_TZ_DB=ON to ensure this change gets tested
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc issue #142: [ORC-218] Cache timezone information in the library.
Posted by omalley <gi...@git.apache.org>.
Github user omalley commented on the issue:
https://github.com/apache/orc/pull/142
You may find the docker scripts useful for testing in other linux platforms.
.. Owen
On Mon, Aug 7, 2017 at 10:06 AM, Martin Grund <no...@github.com>
wrote:
> There's some funky stuff that's going on with the way how the code get's
> compiled in Travis, that I need to reproduce. The unit tests with the same
> combination of parameters works for me locally. Still working on it.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/orc/pull/142#issuecomment-320721828>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AAMmyICAV0Gst3DO2iVUUY4Bn_KbDQH8ks5sV0QVgaJpZM4OjHFO>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #142: [ORC-218] Cache timezone information in the library.
Posted by majetideepak <gi...@git.apache.org>.
Github user majetideepak commented on a diff in the pull request:
https://github.com/apache/orc/pull/142#discussion_r129886305
--- Diff: c++/src/CMakeLists.txt ---
@@ -125,6 +125,83 @@ include_directories (
${LZ4_INCLUDE_DIRS}
)
+# To avoid reading the Timezone database from disk, we load the file during the
+# build and inject them as binary into the library. This can increase the size
+# of library but avoids doing system calls at runtime. The behavior can be
+# disabled by using the CMake flag NO_EMBEDDED_TZ_DB
+if (NOT DEFINED EMBEDDED_TZ_DB)
+ set(EMBEDDED_TZ_DB ON)
--- End diff --
We won't need the ``if`` check then
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #142: [ORC-218] Cache timezone information in the library.
Posted by grundprinzip <gi...@git.apache.org>.
Github user grundprinzip commented on a diff in the pull request:
https://github.com/apache/orc/pull/142#discussion_r131188310
--- Diff: c++/src/CMakeLists.txt ---
@@ -125,6 +125,83 @@ include_directories (
${LZ4_INCLUDE_DIRS}
)
+# To avoid reading the Timezone database from disk, we load the file during the
+# build and inject them as binary into the library. This can increase the size
+# of library but avoids doing system calls at runtime. The behavior can be
+# disabled by using the CMake flag NO_EMBEDDED_TZ_DB
+if (NOT DEFINED EMBEDDED_TZ_DB)
+ set(EMBEDDED_TZ_DB ON)
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #142: [ORC-218] Cache timezone information in the library.
Posted by majetideepak <gi...@git.apache.org>.
Github user majetideepak commented on a diff in the pull request:
https://github.com/apache/orc/pull/142#discussion_r129886934
--- Diff: c++/src/CMakeLists.txt ---
@@ -125,6 +125,83 @@ include_directories (
${LZ4_INCLUDE_DIRS}
)
+# To avoid reading the Timezone database from disk, we load the file during the
+# build and inject them as binary into the library. This can increase the size
+# of library but avoids doing system calls at runtime. The behavior can be
+# disabled by using the CMake flag NO_EMBEDDED_TZ_DB
+if (NOT DEFINED EMBEDDED_TZ_DB)
+ set(EMBEDDED_TZ_DB ON)
+endif()
+if (EMBEDDED_TZ_DB)
+ message(STATUS "Caching of timezone data enabled.")
+ set(DEFAULT_TIMEZONE_DIR "/usr/share/zoneinfo")
+
+ # Check if zone1970.tab is present, otherwise select zone.tab
+ # Parse zone tab to extract all availble zones
+ execute_process(COMMAND find ${DEFAULT_TIMEZONE_DIR} -type f
+ COMMAND grep -v "posix"
+ COMMAND grep -v "right"
+ COMMAND grep -v "tab"
+ RESULT_VARIABLE TZ_LIST_RC
+ OUTPUT_VARIABLE TZ_LIST_OUTPUT
+ ERROR_VARIABLE TZ_ERROR)
+
+ if (TZ_LIST_RC)
+ message(FATAL_ERROR "Could not parse Timezone database.")
+ endif()
+
+ set(FILE2ARRAY "${CMAKE_CURRENT_SOURCE_DIR}/../file2array.sh")
+
+ # For each zone generate a target
+ # Add each target to dependency on Timezone.cc
+ set(TZ_DEPS "")
--- End diff --
hoist this initialization out of the `if (EMBEDDED_TZ_DB)`?
That way we can safely add it as a dependency to orc library on line 214
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---