You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by "Brian Daugavietis (LCL)" <Br...@loblaw.ca> on 2022/04/22 20:18:27 UTC
[Question] Beam DropFields PTransform automatically unnesting remaining fields
Hi,
I have a question about the behavior of the beam built-in DropFields PTransform. If you drop only some fields in a nested row then the other fields in the same nested row are automatically unnested. For example, if a Row originally has columns "A.b", "A.c" and "B.c" and if I use DropField on field "A.b" then the DropField transform will automatically change field "A.c" to "c" in the resulting Row. Meanwhile field "B.c" will remain "B.c". This behavior seems unintuitive. Why should dropping a field require unnesting other fields? Is there any way to avoid this from happening or do I have to write my own PTransform to drop fields without changing the nesting of other fields?
Thanks,
Brian
This email message is confidential, may be legally privileged and is intended for the exclusive use of the addressee. If you received this message in error or are not the intended recipient, you should destroy the email message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained. Please inform us of the delivery error by return email. Thank you for your cooperation.
Le pr?sent message ?lectronique est confidentiel et peut ?tre couvert par le secret professionnel. Il est ? l'usage exclusif du destinataire. Si vous recevez ce message par erreur ou si vous n'en ?tes pas le destinataire pr?vu, vous devez d?truire le message et toute pi?ce jointe ou copie et vous ?tes tenu de ne pas conserver, distribuer, divulguer ni utiliser tout renseignement qu'il contient. Veuillez nous informer de toute erreur d'envoi en r?pondant ? ce message. Merci de votre collaboration.
Re: [Question] Beam DropFields PTransform automatically unnesting remaining fields
Posted by Brian Hulette <bh...@google.com>.
Shoot. I went ahead and filed
https://issues.apache.org/jira/browse/BEAM-14360
On Fri, Apr 22, 2022 at 4:11 PM Reuven Lax <re...@google.com> wrote:
> It was unintentional. DropFields was implemented using an inverse select,
> which is why this is happening. I wonder if we can easily fix this by using
> withFieldNameAs, but I'm not sure offhand whether that will work. If not,
> then we'll have to reimplement DropFields.
>
> On Fri, Apr 22, 2022 at 3:59 PM Brian Hulette <bh...@google.com> wrote:
>
>> Hi Brian,
>> I agree that behavior feels surprising. I thought this could be a bug,
>> but it looks like we are actually verifying this behavior [1]. @Reuven
>> Lax <re...@google.com> is this intentional?
>>
>> Brian (Hulette)
>>
>> [1]
>> https://github.com/apache/beam/blob/2c18ce0ccd7705473aa9ecc443dcdbe223dd9449/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/DropFieldsTest.java#L103
>>
>> On Fri, Apr 22, 2022 at 1:19 PM Brian Daugavietis (LCL) <
>> Brian.Daugavietis@loblaw.ca> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> I have a question about the behavior of the beam built-in DropFields
>>> PTransform. If you drop only some fields in a nested row then the other
>>> fields in the same nested row are automatically unnested. For example, if
>>> a Row originally has columns “A.b”, “A.c” and “B.c” and if I use DropField
>>> on field “A.b” then the DropField transform will automatically change field
>>> “A.c” to “c” in the resulting Row. Meanwhile field “B.c” will remain
>>> “B.c”. This behavior seems unintuitive. Why should dropping a field
>>> require unnesting other fields? Is there any way to avoid this from
>>> happening or do I have to write my own PTransform to drop fields without
>>> changing the nesting of other fields?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Brian
>>>
>>> *This email message is confidential, may be legally privileged and is
>>> intended for the exclusive use of the addressee. If you received this
>>> message in error or are not the intended recipient, you should destroy the
>>> email message and any attachments or copies, and you are prohibited from
>>> retaining, distributing, disclosing or using any information contained.
>>> Please inform us of the delivery error by return email. Thank you for your
>>> cooperation.*
>>>
>>> *Le présent message électronique est confidentiel et peut être couvert
>>> par le secret professionnel. Il est à l’usage exclusif du destinataire. Si
>>> vous recevez ce message par erreur ou si vous n’en êtes pas le destinataire
>>> prévu, vous devez détruire le message et toute pièce jointe ou copie et
>>> vous êtes tenu de ne pas conserver, distribuer, divulguer ni utiliser tout
>>> renseignement qu’il contient. Veuillez nous informer de toute erreur
>>> d’envoi en répondant à ce message. Merci de votre collaboration.*
>>>
>>
Re: [Question] Beam DropFields PTransform automatically unnesting remaining fields
Posted by Reuven Lax <re...@google.com>.
It was unintentional. DropFields was implemented using an inverse select,
which is why this is happening. I wonder if we can easily fix this by using
withFieldNameAs, but I'm not sure offhand whether that will work. If not,
then we'll have to reimplement DropFields.
On Fri, Apr 22, 2022 at 3:59 PM Brian Hulette <bh...@google.com> wrote:
> Hi Brian,
> I agree that behavior feels surprising. I thought this could be a bug, but
> it looks like we are actually verifying this behavior [1]. @Reuven Lax
> <re...@google.com> is this intentional?
>
> Brian (Hulette)
>
> [1]
> https://github.com/apache/beam/blob/2c18ce0ccd7705473aa9ecc443dcdbe223dd9449/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/DropFieldsTest.java#L103
>
> On Fri, Apr 22, 2022 at 1:19 PM Brian Daugavietis (LCL) <
> Brian.Daugavietis@loblaw.ca> wrote:
>
>> Hi,
>>
>>
>>
>> I have a question about the behavior of the beam built-in DropFields
>> PTransform. If you drop only some fields in a nested row then the other
>> fields in the same nested row are automatically unnested. For example, if
>> a Row originally has columns “A.b”, “A.c” and “B.c” and if I use DropField
>> on field “A.b” then the DropField transform will automatically change field
>> “A.c” to “c” in the resulting Row. Meanwhile field “B.c” will remain
>> “B.c”. This behavior seems unintuitive. Why should dropping a field
>> require unnesting other fields? Is there any way to avoid this from
>> happening or do I have to write my own PTransform to drop fields without
>> changing the nesting of other fields?
>>
>>
>>
>> Thanks,
>>
>> Brian
>>
>> *This email message is confidential, may be legally privileged and is
>> intended for the exclusive use of the addressee. If you received this
>> message in error or are not the intended recipient, you should destroy the
>> email message and any attachments or copies, and you are prohibited from
>> retaining, distributing, disclosing or using any information contained.
>> Please inform us of the delivery error by return email. Thank you for your
>> cooperation.*
>>
>> *Le présent message électronique est confidentiel et peut être couvert
>> par le secret professionnel. Il est à l’usage exclusif du destinataire. Si
>> vous recevez ce message par erreur ou si vous n’en êtes pas le destinataire
>> prévu, vous devez détruire le message et toute pièce jointe ou copie et
>> vous êtes tenu de ne pas conserver, distribuer, divulguer ni utiliser tout
>> renseignement qu’il contient. Veuillez nous informer de toute erreur
>> d’envoi en répondant à ce message. Merci de votre collaboration.*
>>
>
Re: [Question] Beam DropFields PTransform automatically unnesting remaining fields
Posted by Brian Hulette <bh...@google.com>.
Hi Brian,
I agree that behavior feels surprising. I thought this could be a bug, but
it looks like we are actually verifying this behavior [1]. @Reuven Lax
<re...@google.com> is this intentional?
Brian (Hulette)
[1]
https://github.com/apache/beam/blob/2c18ce0ccd7705473aa9ecc443dcdbe223dd9449/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/DropFieldsTest.java#L103
On Fri, Apr 22, 2022 at 1:19 PM Brian Daugavietis (LCL) <
Brian.Daugavietis@loblaw.ca> wrote:
> Hi,
>
>
>
> I have a question about the behavior of the beam built-in DropFields
> PTransform. If you drop only some fields in a nested row then the other
> fields in the same nested row are automatically unnested. For example, if
> a Row originally has columns “A.b”, “A.c” and “B.c” and if I use DropField
> on field “A.b” then the DropField transform will automatically change field
> “A.c” to “c” in the resulting Row. Meanwhile field “B.c” will remain
> “B.c”. This behavior seems unintuitive. Why should dropping a field
> require unnesting other fields? Is there any way to avoid this from
> happening or do I have to write my own PTransform to drop fields without
> changing the nesting of other fields?
>
>
>
> Thanks,
>
> Brian
>
> *This email message is confidential, may be legally privileged and is
> intended for the exclusive use of the addressee. If you received this
> message in error or are not the intended recipient, you should destroy the
> email message and any attachments or copies, and you are prohibited from
> retaining, distributing, disclosing or using any information contained.
> Please inform us of the delivery error by return email. Thank you for your
> cooperation.*
>
> *Le présent message électronique est confidentiel et peut être couvert par
> le secret professionnel. Il est à l’usage exclusif du destinataire. Si vous
> recevez ce message par erreur ou si vous n’en êtes pas le destinataire
> prévu, vous devez détruire le message et toute pièce jointe ou copie et
> vous êtes tenu de ne pas conserver, distribuer, divulguer ni utiliser tout
> renseignement qu’il contient. Veuillez nous informer de toute erreur
> d’envoi en répondant à ce message. Merci de votre collaboration.*
>