You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ponymail.apache.org by sebb <se...@gmail.com> on 2021/11/26 14:21:27 UTC
Re: [incubator-ponymail-foal] branch master updated: Strip superfluous whitespace from vital headers
That does not work for headers with multiple values
On Fri, 26 Nov 2021 at 13:50, <hu...@apache.org> wrote:
>
> This is an automated email from the ASF dual-hosted git repository.
>
> humbedooh pushed a commit to branch master
> in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
>
>
> The following commit(s) were added to refs/heads/master by this push:
> new 146f15c Strip superfluous whitespace from vital headers
> 146f15c is described below
>
> commit 146f15cc5a97d741bcfcd5a6584f82a49490d053
> Author: Daniel Gruno <hu...@apache.org>
> AuthorDate: Fri Nov 26 14:50:20 2021 +0100
>
> Strip superfluous whitespace from vital headers
> ---
> tools/archiver.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tools/archiver.py b/tools/archiver.py
> index 8256fac..b3859ea 100755
> --- a/tools/archiver.py
> +++ b/tools/archiver.py
> @@ -475,7 +475,7 @@ class Archiver(object): # N.B. Also used by import-mbox.py
> )
> else:
> hval += t[0].decode(t[1], errors="ignore")
> - msg_metadata[key] = hval
> + msg_metadata[key] = hval.strip()
> except Exception as err:
> print("Could not decode headers, ignoring..: %s" % err)
> message_date = None
Re: [incubator-ponymail-foal] branch master updated: Strip superfluous whitespace from vital headers
Posted by Daniel Gruno <hu...@apache.org>.
On 26/11/2021 16.23, sebb wrote:
> On Fri, 26 Nov 2021 at 14:58, Daniel Gruno <hu...@apache.org> wrote:
>>
>> On 26/11/2021 15.47, sebb wrote:
>>> On Fri, 26 Nov 2021 at 14:37, Daniel Gruno <hu...@apache.org> wrote:
>>>>
>>>> On 26/11/2021 15.21, sebb wrote:
>>>>> That does not work for headers with multiple values
>>>>
>>>> The four headers in question (to, from, subject, message-id) should
>>>> never have multiple values, right?
>>>
>>> To and From (as long as Sender is provided) can both have multiple values.
>>>
>>> As can other headers (e.g. irt) that are stored in the mbox index.
>>
>> Right, what I should say is, they don't have multiple values in a way
>> that matters to our database. Trimming the excess whitespace is therefor
>> fine the way it currently is.
>
> That remains to be seen; I am not convinced.
>
>> For headers where we may care about multiple values (IRT and
>> References), we need to figure out how to split those properly.
>
>> ElasticSearch doesn't care if a value is a single string or an array of
>> strings, so the mappings and search should allow for an updated version
>> of those two fields without any changes to the db.
>
> Surely that depends on the mapping type and the sort of searches?
According to
https://www.elastic.co/guide/en/elasticsearch/reference/current/array.html
any string is essentially a 1-length array of strings and so you can
have a mix of strings and array of strings as long as they are all the
same data type (string vs number etc)
>
>>>
>>> Indeed even single-valued headers can have embedded new-lines, so
>>> fixing the ends of the string is not sufficient.
>>>
>>> Removing line-wraps is tricky to do correctly, so the code should use
>>> the appropriate email methods to improve compatibility with the RFCs.
>>>
>>>>>
>>>>> On Fri, 26 Nov 2021 at 13:50, <hu...@apache.org> wrote:
>>>>>>
>>>>>> This is an automated email from the ASF dual-hosted git repository.
>>>>>>
>>>>>> humbedooh pushed a commit to branch master
>>>>>> in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
>>>>>>
>>>>>>
>>>>>> The following commit(s) were added to refs/heads/master by this push:
>>>>>> new 146f15c Strip superfluous whitespace from vital headers
>>>>>> 146f15c is described below
>>>>>>
>>>>>> commit 146f15cc5a97d741bcfcd5a6584f82a49490d053
>>>>>> Author: Daniel Gruno <hu...@apache.org>
>>>>>> AuthorDate: Fri Nov 26 14:50:20 2021 +0100
>>>>>>
>>>>>> Strip superfluous whitespace from vital headers
>>>>>> ---
>>>>>> tools/archiver.py | 2 +-
>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/tools/archiver.py b/tools/archiver.py
>>>>>> index 8256fac..b3859ea 100755
>>>>>> --- a/tools/archiver.py
>>>>>> +++ b/tools/archiver.py
>>>>>> @@ -475,7 +475,7 @@ class Archiver(object): # N.B. Also used by import-mbox.py
>>>>>> )
>>>>>> else:
>>>>>> hval += t[0].decode(t[1], errors="ignore")
>>>>>> - msg_metadata[key] = hval
>>>>>> + msg_metadata[key] = hval.strip()
>>>>>> except Exception as err:
>>>>>> print("Could not decode headers, ignoring..: %s" % err)
>>>>>> message_date = None
>>>>
>>
Re: [incubator-ponymail-foal] branch master updated: Strip superfluous whitespace from vital headers
Posted by sebb <se...@gmail.com>.
On Fri, 26 Nov 2021 at 14:58, Daniel Gruno <hu...@apache.org> wrote:
>
> On 26/11/2021 15.47, sebb wrote:
> > On Fri, 26 Nov 2021 at 14:37, Daniel Gruno <hu...@apache.org> wrote:
> >>
> >> On 26/11/2021 15.21, sebb wrote:
> >>> That does not work for headers with multiple values
> >>
> >> The four headers in question (to, from, subject, message-id) should
> >> never have multiple values, right?
> >
> > To and From (as long as Sender is provided) can both have multiple values.
> >
> > As can other headers (e.g. irt) that are stored in the mbox index.
>
> Right, what I should say is, they don't have multiple values in a way
> that matters to our database. Trimming the excess whitespace is therefor
> fine the way it currently is.
That remains to be seen; I am not convinced.
> For headers where we may care about multiple values (IRT and
> References), we need to figure out how to split those properly.
> ElasticSearch doesn't care if a value is a single string or an array of
> strings, so the mappings and search should allow for an updated version
> of those two fields without any changes to the db.
Surely that depends on the mapping type and the sort of searches?
> >
> > Indeed even single-valued headers can have embedded new-lines, so
> > fixing the ends of the string is not sufficient.
> >
> > Removing line-wraps is tricky to do correctly, so the code should use
> > the appropriate email methods to improve compatibility with the RFCs.
> >
> >>>
> >>> On Fri, 26 Nov 2021 at 13:50, <hu...@apache.org> wrote:
> >>>>
> >>>> This is an automated email from the ASF dual-hosted git repository.
> >>>>
> >>>> humbedooh pushed a commit to branch master
> >>>> in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
> >>>>
> >>>>
> >>>> The following commit(s) were added to refs/heads/master by this push:
> >>>> new 146f15c Strip superfluous whitespace from vital headers
> >>>> 146f15c is described below
> >>>>
> >>>> commit 146f15cc5a97d741bcfcd5a6584f82a49490d053
> >>>> Author: Daniel Gruno <hu...@apache.org>
> >>>> AuthorDate: Fri Nov 26 14:50:20 2021 +0100
> >>>>
> >>>> Strip superfluous whitespace from vital headers
> >>>> ---
> >>>> tools/archiver.py | 2 +-
> >>>> 1 file changed, 1 insertion(+), 1 deletion(-)
> >>>>
> >>>> diff --git a/tools/archiver.py b/tools/archiver.py
> >>>> index 8256fac..b3859ea 100755
> >>>> --- a/tools/archiver.py
> >>>> +++ b/tools/archiver.py
> >>>> @@ -475,7 +475,7 @@ class Archiver(object): # N.B. Also used by import-mbox.py
> >>>> )
> >>>> else:
> >>>> hval += t[0].decode(t[1], errors="ignore")
> >>>> - msg_metadata[key] = hval
> >>>> + msg_metadata[key] = hval.strip()
> >>>> except Exception as err:
> >>>> print("Could not decode headers, ignoring..: %s" % err)
> >>>> message_date = None
> >>
>
Re: [incubator-ponymail-foal] branch master updated: Strip superfluous whitespace from vital headers
Posted by Daniel Gruno <hu...@apache.org>.
On 26/11/2021 15.47, sebb wrote:
> On Fri, 26 Nov 2021 at 14:37, Daniel Gruno <hu...@apache.org> wrote:
>>
>> On 26/11/2021 15.21, sebb wrote:
>>> That does not work for headers with multiple values
>>
>> The four headers in question (to, from, subject, message-id) should
>> never have multiple values, right?
>
> To and From (as long as Sender is provided) can both have multiple values.
>
> As can other headers (e.g. irt) that are stored in the mbox index.
Right, what I should say is, they don't have multiple values in a way
that matters to our database. Trimming the excess whitespace is therefor
fine the way it currently is.
For headers where we may care about multiple values (IRT and
References), we need to figure out how to split those properly.
ElasticSearch doesn't care if a value is a single string or an array of
strings, so the mappings and search should allow for an updated version
of those two fields without any changes to the db.
>
> Indeed even single-valued headers can have embedded new-lines, so
> fixing the ends of the string is not sufficient.
>
> Removing line-wraps is tricky to do correctly, so the code should use
> the appropriate email methods to improve compatibility with the RFCs.
>
>>>
>>> On Fri, 26 Nov 2021 at 13:50, <hu...@apache.org> wrote:
>>>>
>>>> This is an automated email from the ASF dual-hosted git repository.
>>>>
>>>> humbedooh pushed a commit to branch master
>>>> in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
>>>>
>>>>
>>>> The following commit(s) were added to refs/heads/master by this push:
>>>> new 146f15c Strip superfluous whitespace from vital headers
>>>> 146f15c is described below
>>>>
>>>> commit 146f15cc5a97d741bcfcd5a6584f82a49490d053
>>>> Author: Daniel Gruno <hu...@apache.org>
>>>> AuthorDate: Fri Nov 26 14:50:20 2021 +0100
>>>>
>>>> Strip superfluous whitespace from vital headers
>>>> ---
>>>> tools/archiver.py | 2 +-
>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/tools/archiver.py b/tools/archiver.py
>>>> index 8256fac..b3859ea 100755
>>>> --- a/tools/archiver.py
>>>> +++ b/tools/archiver.py
>>>> @@ -475,7 +475,7 @@ class Archiver(object): # N.B. Also used by import-mbox.py
>>>> )
>>>> else:
>>>> hval += t[0].decode(t[1], errors="ignore")
>>>> - msg_metadata[key] = hval
>>>> + msg_metadata[key] = hval.strip()
>>>> except Exception as err:
>>>> print("Could not decode headers, ignoring..: %s" % err)
>>>> message_date = None
>>
Re: [incubator-ponymail-foal] branch master updated: Strip superfluous whitespace from vital headers
Posted by sebb <se...@gmail.com>.
On Fri, 26 Nov 2021 at 14:37, Daniel Gruno <hu...@apache.org> wrote:
>
> On 26/11/2021 15.21, sebb wrote:
> > That does not work for headers with multiple values
>
> The four headers in question (to, from, subject, message-id) should
> never have multiple values, right?
To and From (as long as Sender is provided) can both have multiple values.
As can other headers (e.g. irt) that are stored in the mbox index.
Indeed even single-valued headers can have embedded new-lines, so
fixing the ends of the string is not sufficient.
Removing line-wraps is tricky to do correctly, so the code should use
the appropriate email methods to improve compatibility with the RFCs.
> >
> > On Fri, 26 Nov 2021 at 13:50, <hu...@apache.org> wrote:
> >>
> >> This is an automated email from the ASF dual-hosted git repository.
> >>
> >> humbedooh pushed a commit to branch master
> >> in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
> >>
> >>
> >> The following commit(s) were added to refs/heads/master by this push:
> >> new 146f15c Strip superfluous whitespace from vital headers
> >> 146f15c is described below
> >>
> >> commit 146f15cc5a97d741bcfcd5a6584f82a49490d053
> >> Author: Daniel Gruno <hu...@apache.org>
> >> AuthorDate: Fri Nov 26 14:50:20 2021 +0100
> >>
> >> Strip superfluous whitespace from vital headers
> >> ---
> >> tools/archiver.py | 2 +-
> >> 1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/tools/archiver.py b/tools/archiver.py
> >> index 8256fac..b3859ea 100755
> >> --- a/tools/archiver.py
> >> +++ b/tools/archiver.py
> >> @@ -475,7 +475,7 @@ class Archiver(object): # N.B. Also used by import-mbox.py
> >> )
> >> else:
> >> hval += t[0].decode(t[1], errors="ignore")
> >> - msg_metadata[key] = hval
> >> + msg_metadata[key] = hval.strip()
> >> except Exception as err:
> >> print("Could not decode headers, ignoring..: %s" % err)
> >> message_date = None
>
Re: [incubator-ponymail-foal] branch master updated: Strip superfluous whitespace from vital headers
Posted by Daniel Gruno <hu...@apache.org>.
On 26/11/2021 15.21, sebb wrote:
> That does not work for headers with multiple values
The four headers in question (to, from, subject, message-id) should
never have multiple values, right?
>
> On Fri, 26 Nov 2021 at 13:50, <hu...@apache.org> wrote:
>>
>> This is an automated email from the ASF dual-hosted git repository.
>>
>> humbedooh pushed a commit to branch master
>> in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
>>
>>
>> The following commit(s) were added to refs/heads/master by this push:
>> new 146f15c Strip superfluous whitespace from vital headers
>> 146f15c is described below
>>
>> commit 146f15cc5a97d741bcfcd5a6584f82a49490d053
>> Author: Daniel Gruno <hu...@apache.org>
>> AuthorDate: Fri Nov 26 14:50:20 2021 +0100
>>
>> Strip superfluous whitespace from vital headers
>> ---
>> tools/archiver.py | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/tools/archiver.py b/tools/archiver.py
>> index 8256fac..b3859ea 100755
>> --- a/tools/archiver.py
>> +++ b/tools/archiver.py
>> @@ -475,7 +475,7 @@ class Archiver(object): # N.B. Also used by import-mbox.py
>> )
>> else:
>> hval += t[0].decode(t[1], errors="ignore")
>> - msg_metadata[key] = hval
>> + msg_metadata[key] = hval.strip()
>> except Exception as err:
>> print("Could not decode headers, ignoring..: %s" % err)
>> message_date = None