You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@age.apache.org by "antamel (via GitHub)" <gi...@apache.org> on 2023/10/10 10:35:32 UTC
[PR] Fix sanitizer errors [age]
antamel opened a new pull request, #1266:
URL: https://github.com/apache/age/pull/1266
Hello!
In the build with address and undefined behavior sanitizers, my colleague
Andrew Bille <an...@gmail.com> found several errors
in the current master branch.
The build was made with options like that:
```
CC=gcc-11
CXX=g++-11
CPPFLAGS="-ggdb -O0 -g3 -fno-omit-frame-pointer -fsanitize=address -fsanitize=undefined -fsanitize=alignment -fno-sanitize-recover=all -fno-sanitize=nonnull-attribute -fstack-protector"
LDFLAGS='-fsanitize=address -fsanitize=undefined -static-libasan -static-libubsan'
ASAN_OPTIONS=detect_leaks=0:abort_on_error=1:halt_on_error=1:disable_coredump=0:strict_string_checks=1:check_initialization_order=1:strict_init_order=1:detect_odr_violation=0
```
There are several "load of misaligned address" sanitizer errors in the log
during installcheck tests with PG at REL_15_STABLE (6db384f2f9).
See [master-at-a04fd69b.zip](https://github.com/apache/age/files/12856017/master-at-a04fd69b.zip) attached. (Other errors are masked by this, about them below.)
Some of these errors are not related to age, is a bug of the sanitizer itself
(see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111382).
The other ones i tried to fix with the [Fix sanitizer aligment errors with memcpy.](https://github.com/apache/age/commit/de3633d21e6b879176b3e57ffc377db9aa3edc3f) commit.
After that "misaligned" errors disappeared and other errors became visible.
Please see postmaster.log in the [after-first-fix-at-de3633d2.zip](https://github.com/apache/age/files/12856019/after-first-fix-at-de3633d2.zip).
The second commit [Fix sanitizer "invalid values for type '_Bool' " errors with palloc0.](https://github.com/apache/age/commit/3afbad612251b3e95db1ea0b8daac9344754c2c3) fixes
runtime errors: "load of value xxx, which is not a valid value for type '_Bool'"
while the third [Fix sanitizer heap-buffer-overflow errors during csv files parsing.](https://github.com/apache/age/commit/2840ce1fdbd2027a9a83ffa06334c703fc7e6e24) one
fixes "heap-buffer-overflow on address.." one during csv file parsing.
On applying the first three patches, another error heap-buffer-overflow
became visible. See log in [before-last-fix-at-2840ce1f.zip](https://github.com/apache/age/files/12856022/before-last-fix-at-2840ce1f.zip).
The last commit [Save the metadata representing by the extensible node with the null t…](https://github.com/apache/age/commit/71f445a61d604f4ca8a2bc5f364f4c0077e09ac5)
fixes it add all tests began to pass without errors with
REL_15_STABLE and REL_14_STABLE branches.
Would be happy for any comments and concerns.
With kind regards,
--
Anton A. Melnikov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix sanitizer errors [age]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #1266: Fix sanitizer errors
URL: https://github.com/apache/age/pull/1266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix sanitizer errors [age]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1266:
URL: https://github.com/apache/age/pull/1266#issuecomment-2246599833
This PR is stale because it has been open 60 days with no activity. Remove "Abondoned" label or comment or this will be closed in 14 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix sanitizer errors [age]
Posted by "jrgemignani (via GitHub)" <gi...@apache.org>.
jrgemignani commented on PR #1266:
URL: https://github.com/apache/age/pull/1266#issuecomment-2108483439
@rafsun42 As the original creator just posted this PR and has yet to respond since, should we break this into 2 PRs, one for memcpy, one for palloc0, attribute this PR and implement ourselves?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix sanitizer errors [age]
Posted by "jrgemignani (via GitHub)" <gi...@apache.org>.
jrgemignani commented on code in PR #1266:
URL: https://github.com/apache/age/pull/1266#discussion_r1353047900
##########
src/backend/executor/cypher_set.c:
##########
@@ -255,7 +255,7 @@ static agtype_value *replace_entity_in_path(agtype_value *path,
agtype *prop_agtype;
int i;
- r = palloc(sizeof(agtype_value));
+ r = palloc0(sizeof(agtype_value));
Review Comment:
While it is generally good form to pre-initialize to zero, is it really necessary if the routine that uses it, sets it up properly? In the case of these iterator functions, which are generally slow, it takes up additional cpu cycles. So, I'm not sure this is necessary. Can you justify adding this?
##########
src/backend/utils/load/ag_load_edges.c:
##########
@@ -160,7 +160,7 @@ int create_edges_from_csv_file(char *file_path,
struct csv_parser p;
char buf[1024];
size_t bytes_read;
- unsigned char options = 0;
+ unsigned char options = CSV_APPEND_NULL;
Review Comment:
Please elaborate why. This is not dismissing it, but inquiring why.
##########
src/backend/utils/load/ag_load_labels.c:
##########
@@ -194,7 +194,7 @@ int create_labels_from_csv_file(char *file_path,
struct csv_parser p;
char buf[1024];
size_t bytes_read;
- unsigned char options = 0;
+ unsigned char options = CSV_APPEND_NULL;
Review Comment:
Please elaborate why. This is not dismissing it, but inquiring why.
##########
src/backend/utils/adt/agtype.c:
##########
@@ -9043,7 +9043,7 @@ agtype_value *agtype_composite_to_agtype_value_binary(agtype *a)
errmsg("cannot convert agtype scalar objects to binary agtype_value objects")));
}
- result = palloc(sizeof(agtype_value));
+ result = palloc0(sizeof(agtype_value));
Review Comment:
While it is generally good form to pre-initialize to zero, is it really necessary if the routine that uses it, sets it up properly? In the case of these iterator functions, which are generally slow, it takes up additional cpu cycles. So, I'm not sure this is necessary. Can you justify adding this?
##########
src/backend/utils/adt/agtype_util.c:
##########
@@ -460,7 +460,7 @@ agtype_value *find_agtype_value_from_container(agtype_container *container,
return NULL;
}
- result = palloc(sizeof(agtype_value));
+ result = palloc0(sizeof(agtype_value));
Review Comment:
Similar to the above palloc0 comment. I'm not opposed to adding this in, I'm just not sure that it is necessary.
##########
src/backend/utils/adt/agtype_util.c:
##########
@@ -554,7 +554,7 @@ agtype_value *get_ith_agtype_value_from_container(agtype_container *container,
if (i >= nelements)
return NULL;
- result = palloc(sizeof(agtype_value));
+ result = palloc0(sizeof(agtype_value));
Review Comment:
Similar to the above palloc0 comment. I'm not opposed to adding this in, I'm just not sure that it is necessary.
##########
src/backend/utils/adt/agtype_ext.c:
##########
@@ -195,7 +195,7 @@ static void ag_deserialize_composite(char *base, enum agtype_value_type type,
//offset container by the extended type header
char *container_base = base + AGT_HEADER_SIZE;
- r = palloc(sizeof(agtype_value));
+ r = palloc0(sizeof(agtype_value));
Review Comment:
While it is generally good form to pre-initialize to zero, is it really necessary if the routine that uses it, sets it up properly? In the case of these iterator functions, which are generally slow, it takes up additional cpu cycles. So, I'm not sure this is necessary. Can you justify adding this?
##########
src/backend/utils/adt/agtype_util.c:
##########
@@ -727,7 +727,7 @@ static agtype_value *push_agtype_value_scalar(agtype_parse_state **pstate,
(*pstate)->cont_val.val.object.num_pairs = 0;
(*pstate)->size = 4;
(*pstate)->cont_val.val.object.pairs =
- palloc(sizeof(agtype_pair) * (*pstate)->size);
+ palloc0(sizeof(agtype_pair) * (*pstate)->size);
Review Comment:
Similar to the above palloc0 comment. I'm not opposed to adding this in, I'm just not sure that it is necessary.
##########
src/backend/utils/adt/agtype_util.c:
##########
@@ -1306,7 +1306,7 @@ bool agtype_deep_contains(agtype_iterator **val, agtype_iterator **m_contained)
uint32 j = 0;
/* Make room for all possible values */
- lhs_conts = palloc(sizeof(agtype_value) * num_lhs_elems);
+ lhs_conts = palloc0(sizeof(agtype_value) * num_lhs_elems);
Review Comment:
Similar to the above palloc0 comment. I'm not opposed to adding this in, I'm just not sure that it is necessary.
##########
src/backend/parser/cypher_clause.c:
##########
@@ -6760,7 +6760,7 @@ static FuncExpr *make_clause_func_expr(char *function_name,
*/
outNode(str, clause_information);
- clause_information_const = makeConst(INTERNALOID, -1, InvalidOid, str->len,
+ clause_information_const = makeConst(INTERNALOID, -1, InvalidOid, str->len + 1,
Review Comment:
Please justify why there needs to be an additional +1 to the length?
##########
src/backend/utils/adt/agtype_util.c:
##########
@@ -716,7 +716,7 @@ static agtype_value *push_agtype_value_scalar(agtype_parse_state **pstate,
(*pstate)->size = 4;
}
(*pstate)->cont_val.val.array.elems =
- palloc(sizeof(agtype_value) * (*pstate)->size);
+ palloc0(sizeof(agtype_value) * (*pstate)->size);
Review Comment:
Similar to the above palloc0 comment. I'm not opposed to adding this in, I'm just not sure that it is necessary.
##########
src/backend/utils/adt/agtype_ext.c:
##########
@@ -59,7 +59,7 @@ bool ag_serialize_extended_type(StringInfo buffer, agtentry *agtentry,
/* copy in the int_value data */
numlen = sizeof(int64);
offset = reserve_from_buffer(buffer, numlen);
- *((int64 *)(buffer->data + offset)) = scalar_val->val.int_value;
+ memcpy( buffer->data + offset, &scalar_val->val.int_value, numlen);
Review Comment:
I don't see why this is necessary, maybe it is just me, could you please elaborate?
##########
src/backend/utils/adt/agtype_ext.c:
##########
@@ -156,12 +156,12 @@ void ag_deserialize_extended_type(char *base_addr, uint32 offset,
{
case AGT_HEADER_INTEGER:
result->type = AGTV_INTEGER;
- result->val.int_value = *((int64 *)(base + AGT_HEADER_SIZE));
+ memcpy(&result->val.int_value, base + AGT_HEADER_SIZE, sizeof(int64));
Review Comment:
I don't see why this is necessary, maybe it is just me, could you please elaborate?
##########
src/backend/utils/adt/agtype_ext.c:
##########
@@ -70,7 +70,7 @@ bool ag_serialize_extended_type(StringInfo buffer, agtentry *agtentry,
/* copy in the float_value data */
numlen = sizeof(scalar_val->val.float_value);
offset = reserve_from_buffer(buffer, numlen);
- *((float8 *)(buffer->data + offset)) = scalar_val->val.float_value;
+ memcpy(buffer->data + offset, &scalar_val->val.int_value, numlen);
Review Comment:
I don't see why this is necessary, maybe it is just me, could you please elaborate?
##########
src/backend/utils/adt/agtype_ext.c:
##########
@@ -156,12 +156,12 @@ void ag_deserialize_extended_type(char *base_addr, uint32 offset,
{
case AGT_HEADER_INTEGER:
result->type = AGTV_INTEGER;
- result->val.int_value = *((int64 *)(base + AGT_HEADER_SIZE));
+ memcpy(&result->val.int_value, base + AGT_HEADER_SIZE, sizeof(int64));
break;
case AGT_HEADER_FLOAT:
result->type = AGTV_FLOAT;
- result->val.float_value = *((float8 *)(base + AGT_HEADER_SIZE));
+ memcpy(&result->val.float_value, base + AGT_HEADER_SIZE, sizeof(float8));
Review Comment:
I don't see why this is necessary, maybe it is just me, could you please elaborate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix sanitizer errors [age]
Posted by "rafsun42 (via GitHub)" <gi...@apache.org>.
rafsun42 commented on PR #1266:
URL: https://github.com/apache/age/pull/1266#issuecomment-1947518753
+1 for the palloc0 changes as PostgreSQL recommends it as well:
https://www.postgresql.org/docs/current/xfunc-c.html
> Always zero the bytes of your structures using memset (or allocate them with palloc0 in the first place). Even if you assign to each field of your structure, there might be alignment padding (holes in the structure) that contain garbage values. Without this, it's difficult to support hash indexes or hash joins, as you must pick out only the significant bits of your data structure to compute a hash. The planner also sometimes relies on comparing constants via bitwise equality, so you can get undesirable planning results if logically-equivalent values aren't bitwise equal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix sanitizer errors [age]
Posted by "rafsun42 (via GitHub)" <gi...@apache.org>.
rafsun42 commented on PR #1266:
URL: https://github.com/apache/age/pull/1266#issuecomment-2130389089
@jrgemignani Sounds good. I am fine with doing either of palloc0 or memcpy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix sanitizer errors [age]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1266:
URL: https://github.com/apache/age/pull/1266#issuecomment-2105397168
This PR is stale because it has been open 45 days with no activity. Remove "Abondoned" label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix sanitizer errors [age]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1266:
URL: https://github.com/apache/age/pull/1266#issuecomment-2274572541
This PR was closed because it has been stalled for further 14 days with no activity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org