You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Matthew Jacobs (JIRA)" <ji...@apache.org> on 2017/07/13 23:09:00 UTC
[jira] [Created] (IMPALA-5664) Unix time to timestamp conversions
may crash impala (boost exception)
Matthew Jacobs created IMPALA-5664:
--------------------------------------
Summary: Unix time to timestamp conversions may crash impala (boost exception)
Key: IMPALA-5664
URL: https://issues.apache.org/jira/browse/IMPALA-5664
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.7.0
Reporter: Matthew Jacobs
Priority: Critical
Unix time to timestamp conversions with dates before 1400 may crash Impala in some cases.
E.g.
{code}
[localhost:21000] > select cast(-17987443200-0.1 as timestamp);
Query: select cast(-17987443200-0.1 as timestamp)
Query submitted at: 2017-07-13 15:18:05 (Coordinator: http://mj-desktop.ca.cloudera.com:25000)
Error communicating with impalad: TSocket read 0 bytes
(gdb) bt
#0 0x00007f3a2a29dc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f3a2a2a1028 in __GI_abort () at abort.c:89
#2 0x00007f3a2adc9cbd in __gnu_cxx::__verbose_terminate_handler () at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/vterminate.cc:95
#3 0x00007f3a2adc7d46 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_terminate.cc:47
#4 0x00007f3a2adc7d91 in std::terminate () at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_terminate.cc:57
#5 0x00007f3a2adc7fa8 in __cxxabiv1::__cxa_throw (obj=0x964abc0,
tinfo=0x23825e0 <typeinfo for boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::gregorian::bad_year> >>,
dest=0x7daed0 <boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::gregorian::bad_year> >::~clone_impl()>)
at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_throw.cc:87
#6 0x00000000007d6596 in boost::throw_exception<boost::gregorian::bad_year> (e=...) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/throw_exception.hpp:69
#7 0x00000000007d1842 in boost::CV::simple_exception_policy<unsigned short, (unsigned short)1400, (unsigned short)9999, boost::gregorian::bad_year>::on_error ()
at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/constrained_value.hpp:110
#8 0x00000000007cc18e in boost::CV::constrained_value<boost::CV::simple_exception_policy<unsigned short, (unsigned short)1400, (unsigned short)9999, boost::gregorian::bad_year> >::assign (this=0x7f39d94ff7c0, value=1399) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/constrained_value.hpp:69
#9 0x00000000007c5c4a in boost::CV::constrained_value<boost::CV::simple_exception_policy<unsigned short, (unsigned short)1400, (unsigned short)9999, boost::gregorian::bad_year> >::constrained_value (this=0x7f39d94ff7c0, value=1399) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/constrained_value.hpp:48
#10 0x00000000007c19aa in boost::gregorian::greg_year::greg_year (this=0x7f39d94ff7c0, year=1399)
at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian/greg_year.hpp:41
#11 0x00007f3a302948de in boost::date_time::gregorian_calendar_base<boost::date_time::year_month_day_base<boost::gregorian::greg_year, boost::gregorian::greg_month, boost::gregorian::greg_day>, unsigned int>::from_day_number (dayNumber=2232399) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian_calendar.ipp:122
#12 0x00007f3a302985ef in boost::date_time::date<boost::gregorian::date, boost::gregorian::gregorian_calendar, boost::gregorian::date_duration>::year_month_day (
this=0x7f39d94ff870) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/date.hpp:99
#13 0x00007f3a30298afb in boost::date_time::date_formatter<boost::gregorian::date, boost::date_time::iso_extended_format<char>, char>::date_to_string (d=...)
at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/date_formatting.hpp:125
#14 0x00007f3a3029867f in boost::gregorian::to_iso_extended_string_type<char> (d=...)
at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian/formatters.hpp:79
#15 0x00007f3a302983d7 in boost::gregorian::to_iso_extended_string (d=...) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian/formatters.hpp:85
#16 0x00007f3a302980d1 in impala::TimestampValue::ToString (this=0x964aae0) at /home/mj/dev/Impala/be/src/runtime/timestamp-value.cc:178
#17 0x00007f3a30297db9 in impala::operator<< (os=..., timestamp_value=...) at /home/mj/dev/Impala/be/src/runtime/timestamp-value.cc:112
#18 0x00007f3a3025feb2 in impala::RawValue::PrintValue (value=0x964aae0, type=..., scale=-1, stream=0x7f39d94ffc30) at /home/mj/dev/Impala/be/src/runtime/raw-value.cc:286
#19 0x00007f3a3025f03a in impala::RawValue::PrintValue (value=0x964aae0, type=..., scale=-1, str=0x7f39d94fffc0) at /home/mj/dev/Impala/be/src/runtime/raw-value.cc:108
#20 0x00007f3a2e849bd8 in SetTColumnValue (value=0x964aae0, type=..., col_val=0x7f39d94fffa0) at /home/mj/dev/Impala/be/src/service/fe-support.cc:143
#21 0x00007f3a2e84a73d in Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow (env=0x53f09e8, caller_class=0x7f39d9500ae8, thrift_expr_batch=0x7f39d9500b00,
thrift_query_ctx_bytes=0x7f39d9500af8) at /home/mj/dev/Impala/be/src/service/fe-support.cc:236
{code}
The same issue may occur in BE execution as well (as opposed to FeSupport above). E.g. similarly:
{code}
[localhost:21001] > select cast(-17987443200-(tinyint_col/10.0) as timestamp) from alltypestiny where id < 2;
Query: select cast(-17987443200-(tinyint_col/10.0) as timestamp) from alltypestiny where id < 2
Query submitted at: 2017-07-13 15:27:20 (Coordinator: http://mj-desktop.ca.cloudera.com:25001)
Query progress can be monitored at: http://mj-desktop.ca.cloudera.com:25001/query_plan?query_id=554eb0b7787304cb:f6be064200000000
#0 0x00007fb90618cc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fb906190028 in __GI_abort () at abort.c:89
#2 0x00007fb906cb8cbd in __gnu_cxx::__verbose_terminate_handler () at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/vterminate.cc:95
#3 0x00007fb906cb6d46 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_terminate.cc:47
#4 0x00007fb906cb6d91 in std::terminate () at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_terminate.cc:57
#5 0x00007fb906cb6fa8 in __cxxabiv1::__cxa_throw (obj=0xa371dc0,
tinfo=0x23825e0 <typeinfo for boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::gregorian::bad_year> >>,
dest=0x7daed0 <boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::gregorian::bad_year> >::~clone_impl()>)
at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_throw.cc:87
#6 0x00000000007d6596 in boost::throw_exception<boost::gregorian::bad_year> (e=...) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/throw_exception.hpp:69
#7 0x00000000007d1842 in boost::CV::simple_exception_policy<unsigned short, (unsigned short)1400, (unsigned short)9999, boost::gregorian::bad_year>::on_error ()
at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/constrained_value.hpp:110
#8 0x00000000007cc18e in boost::CV::constrained_value<boost::CV::simple_exception_policy<unsigned short, (unsigned short)1400, (unsigned short)9999, boost::gregorian::bad_year> >::assign (this=0x7fb872b63f20, value=1399) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/constrained_value.hpp:69
#9 0x00000000007c5c4a in boost::CV::constrained_value<boost::CV::simple_exception_policy<unsigned short, (unsigned short)1400, (unsigned short)9999, boost::gregorian::bad_year> >::constrained_value (this=0x7fb872b63f20, value=1399) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/constrained_value.hpp:48
#10 0x00000000007c19aa in boost::gregorian::greg_year::greg_year (this=0x7fb872b63f20, year=1399)
at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian/greg_year.hpp:41
#11 0x00007fb90c1838de in boost::date_time::gregorian_calendar_base<boost::date_time::year_month_day_base<boost::gregorian::greg_year, boost::gregorian::greg_month, boost::gregorian::greg_day>, unsigned int>::from_day_number (dayNumber=2232399) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian_calendar.ipp:122
#12 0x00007fb90c1875ef in boost::date_time::date<boost::gregorian::date, boost::gregorian::gregorian_calendar, boost::gregorian::date_duration>::year_month_day (
this=0x7fb872b63fd0) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/date.hpp:99
#13 0x00007fb90c187afb in boost::date_time::date_formatter<boost::gregorian::date, boost::date_time::iso_extended_format<char>, char>::date_to_string (d=...)
at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/date_formatting.hpp:125
#14 0x00007fb90c18767f in boost::gregorian::to_iso_extended_string_type<char> (d=...)
at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian/formatters.hpp:79
#15 0x00007fb90c1873d7 in boost::gregorian::to_iso_extended_string (d=...) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/date_time/gregorian/formatters.hpp:85
#16 0x00007fb90c1870d1 in impala::TimestampValue::ToString (this=0xa371ce0) at /home/mj/dev/Impala/be/src/runtime/timestamp-value.cc:178
#17 0x00007fb90c186db9 in impala::operator<< (os=..., timestamp_value=...) at /home/mj/dev/Impala/be/src/runtime/timestamp-value.cc:112
#18 0x00007fb90c14eeb2 in impala::RawValue::PrintValue (value=0xa371ce0, type=..., scale=-1, stream=0x7fb872b64390) at /home/mj/dev/Impala/be/src/runtime/raw-value.cc:286
#19 0x00007fb90a87905d in impala::AsciiQueryResultSet::AddOneRow (this=0x8e65bc0, col_values=..., scales=...) at /home/mj/dev/Impala/be/src/service/query-result-set.cc:172
#20 0x00007fb90b9ce97a in impala::PlanRootSink::Send (this=0x92d67e0, state=0x5870000, batch=0x8ca68f0) at /home/mj/dev/Impala/be/src/exec/plan-root-sink.cc:97
#21 0x00007fb90c10bf77 in impala::FragmentInstanceState::ExecInternal (this=0x940b480) at /home/mj/dev/Impala/be/src/runtime/fragment-instance-state.cc:276
#22 0x00007fb90c10964f in impala::FragmentInstanceState::Exec (this=0x940b480) at /home/mj/dev/Impala/be/src/runtime/fragment-instance-state.cc:89
#23 0x00007fb90c132e28 in impala::QueryState::ExecFInstance (this=0x885f800, fis=0x940b480) at /home/mj/dev/Impala/be/src/runtime/query-state.cc:321
#24 0x00007fb90c131b3a in impala::QueryState::<lambda()>::operator()(void) const (__closure=0x7fb872b64d28) at /home/mj/dev/Impala/be/src/runtime/query-state.cc:295
#25 0x00007fb90c133805 in boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::<lambda()>, void>::invoke(boost::detail::function::function_buffer &) (function_obj_ptr=...) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
#26 0x00007fb90c6085da in boost::function0<void>::operator() (this=0x7fb872b64d20)
at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
#27 0x00007fb90c605c67 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name=..., category=...,
functor=..., thread_started=0x7fb86e35ad80) at /home/mj/dev/Impala/be/src/util/thread.cc:330
#28 0x00007fb90c60f1fc in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0xbd4f5c0,
f=@0xbd4f5b8: 0x7fb90c605948 <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...)
at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
#29 0x00007fb90c60f13f in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (
this=0xbd4f5b8) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
#30 0x00007fb90c60f102 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() (this=0xbd4f400) at /home/mj/dev/Impala/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
#31 0x000000000087d35a in thread_proxy ()
#32 0x00007fb906527184 in start_thread (arg=0x7fb872b65700) at pthread_create.c:312
#33 0x00007fb906253ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
{code}
It looks like the issue is that code that uses boost time_duration addition/subtraction may create a ptime with an invalid date, yet the ptime/date does not indicate that it is {{is_special()}} that we normally check (e.g. in {{TimestampValue::HasDate()}}.
The code in timestamp-functions-ir.cc {{AddSub}} has an obscure call to {{ptime::date()::year()}} which isn't explained, but looking at the git history there is a relevant comment:
From commit cf60967b:
{code}
-/// The AddInterval() functions provide a unified interface for adding intervals of all
-/// types. To subtract, the 'interval' can be negative. The default template below only
-/// handles time intervals (hours, seconds, etc). Date intervals (days, weeks, etc) need
-/// to be specified individually.
-template <typename Interval>
-inline TimestampVal AddInterval(const TimestampValue& timestamp, int64_t interval) {
- ptime temp;
- timestamp.ToPtime(&temp);
- temp += Interval(interval);
- temp.date().year(); // Forces validation on the year (which will throw).
- const TimestampValue value(temp);
- TimestampVal val;
- value.ToTimestampVal(&val);
- return val;
{code}
A bit more digging shows this related comment:
{code}
// Adding/subtracting boost::gregorian::dates can throw if the result exceeds the
// min/max supported dates. (Sometimes the exception is thrown lazily and calling an
// accessor functions is needed to trigger validation.)
{code}
It seems we need additional validation after using date addition/subtraction. It seems like calling {{year()}} w/in a try/catch block may work, though hopefully there's a better way to do this.
It looks like the FromUnixTime functions that take subseconds, microseconds, and nanoseconds may all be susceptible.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)