You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Sonam Ramchand <so...@venturedive.com> on 2021/03/01 08:37:24 UTC
Random outputs for ARRAY_CONCAT_AGG fn zetasql
Hi Devs,
I have implemented the ARRAY_CONCAT_AGG function for zetasql dialect. I am
trying to validate the test as:
@Test
public void testArrayConcatAggZetasql() {
String sql =
"SELECT ARRAY_CONCAT_AGG(x) AS array_concat_agg FROM (SELECT [1,
2, 3, 4] AS x UNION ALL SELECT [5, 6] UNION ALL SELECT [7, 8, 9])";
ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline,
beamRelNode);
Schema schema = Schema.builder().addArrayField("array_field",
FieldType.INT64).build();
PAssert.that(stream)
.containsInAnyOrder(
Row.withSchema(schema).addArray(1L, 2L, 3L, 4L, 5L, 6L, 7L,
8L, 9L).build());
pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
}
Expected Output is: 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L.
But I am getting randomly different outputs:
1. 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L
2. 5L, 6L, 7L, 8L, 9L, 1L, 2L, 3L, 4L
3. 7L, 8L, 9L, 5L, 6L, 1L, 2L, 3L, 4L
As per my understanding, it is because of containsInAnyOrder function. Is
there anything Like:
PAssert.that(stream)
.containsAnyOfThem(
Row.withSchema(schema).addArray(1L, 2L, 3L, 4L, 5L, 6L,
7L, 8L, 9L).build(),
Row.withSchema(schema).addArray(5L, 6L, 7L, 8L, 9L,
1L, 2L, 3L, 4L).build(),
Row.withSchema(schema).addArray(7L, 8L, 9L, 5L, 6L,
1L, 2L, 3L, 4L).build());
I would really appreciate if anyone can help me in knowing how to handle
such scenario in Beam.
Thanks!
--
Regards,
*Sonam*
Software Engineer
Mobile: +92 3088337296
<http://venturedive.com/>
Re: Random outputs for ARRAY_CONCAT_AGG fn zetasql
Posted by Kyle Weaver <kc...@google.com>.
You will be able to access the actual value inside the function you pass to
satisfies. I just meant that you will have to go through a function call.
On Wed, Mar 3, 2021 at 1:58 AM Sonam Ramchand <
sonam.ramchand@venturedive.com> wrote:
> Yeah, that sounds right. But the only thing that confuses me is:
>
> PAssert.that(stream).satisfies(row -> assertThat("array_agg_concat_field", actual ,
> containsInAnyOrder(Arrays.asList(1L,2L,3L,4L,5L,6L))));
>
> How come I can access *actual* here when output is not materialized.
>
> On Tue, Mar 2, 2021 at 10:49 PM Kyle Weaver <kc...@google.com> wrote:
>
>> As you can see from existing tests, Beam doesn't materialize the output
>> array directly. Instead you must use the PAssert API. I agree with Tyson's
>> suggestion to use `satisfies`, which lets you do arbitrary assertions on
>> the output data.
>>
>> On Tue, Mar 2, 2021 at 3:57 AM Sonam Ramchand <
>> sonam.ramchand@venturedive.com> wrote:
>>
>>> Is there any way I can access the output array resulting from the sql
>>> query? Then maybe I can sort and compare both *output array* and *expected
>>> output array *for the test to pass.
>>>
>>> On Tue, Mar 2, 2021 at 12:24 AM Kenneth Knowles <ke...@apache.org> wrote:
>>>
>>>> Yea, the reason is that SQL relations are not ordered. So any ordering
>>>> of [1, 2, 3, 4] and [5, 6] and [7, 8, 9] is possible and correct.
>>>>
>>>> Kenn
>>>>
>>>> On Mon, Mar 1, 2021 at 11:01 AM Tyson Hamilton <ty...@google.com>
>>>> wrote:
>>>>
>>>>> I didn't find anything like that after a brief look. What you could do
>>>>> instead is something like:
>>>>>
>>>>> PAssert.thatSingleton(stream).satisfies( row ->
>>>>> assertThat("array_field containsInAnyOrder", row.getArray("array_field"),
>>>>> containsInAnyOrder(Arrays.asList(...)));
>>>>>
>>>>> using junit/hamcrest matchers. I didn't verify this works myself but
>>>>> it should give you an idea for some next steps.
>>>>>
>>>>>
>>>>> On Mon, Mar 1, 2021 at 12:37 AM Sonam Ramchand <
>>>>> sonam.ramchand@venturedive.com> wrote:
>>>>>
>>>>>> Hi Devs,
>>>>>> I have implemented the ARRAY_CONCAT_AGG function for zetasql dialect.
>>>>>> I am trying to validate the test as:
>>>>>>
>>>>>> @Test
>>>>>> public void testArrayConcatAggZetasql() {
>>>>>> String sql =
>>>>>> "SELECT ARRAY_CONCAT_AGG(x) AS array_concat_agg FROM (SELECT [1, 2, 3, 4] AS x UNION ALL SELECT [5, 6] UNION ALL SELECT [7, 8, 9])";
>>>>>>
>>>>>> ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
>>>>>> BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
>>>>>> PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
>>>>>>
>>>>>> Schema schema = Schema.builder().addArrayField("array_field", FieldType.INT64).build();
>>>>>> PAssert.that(stream)
>>>>>> .containsInAnyOrder(
>>>>>> Row.withSchema(schema).addArray(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L).build());
>>>>>> pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
>>>>>> }
>>>>>>
>>>>>> Expected Output is: 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L.
>>>>>> But I am getting randomly different outputs:
>>>>>> 1. 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L
>>>>>> 2. 5L, 6L, 7L, 8L, 9L, 1L, 2L, 3L, 4L
>>>>>> 3. 7L, 8L, 9L, 5L, 6L, 1L, 2L, 3L, 4L
>>>>>>
>>>>>> As per my understanding, it is because of containsInAnyOrder
>>>>>> function. Is there anything Like:
>>>>>>
>>>>>> PAssert.that(stream)
>>>>>> .containsAnyOfThem(
>>>>>> Row.withSchema(schema).addArray(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L).build(),
>>>>>> Row.withSchema(schema).addArray(5L, 6L, 7L, 8L, 9L, 1L, 2L, 3L, 4L).build(),
>>>>>> Row.withSchema(schema).addArray(7L, 8L, 9L, 5L, 6L, 1L, 2L, 3L, 4L).build());
>>>>>>
>>>>>> I would really appreciate if anyone can help me in knowing how to
>>>>>> handle such scenario in Beam.
>>>>>>
>>>>>> Thanks!
>>>>>> --
>>>>>> Regards,
>>>>>> *Sonam*
>>>>>> Software Engineer
>>>>>> Mobile: +92 3088337296 <+92%20308%208337296>
>>>>>>
>>>>>> <http://venturedive.com/>
>>>>>>
>>>>>
>>>
>>> --
>>>
>>> Regards,
>>> *Sonam*
>>> Software Engineer
>>> Mobile: +92 3088337296 <+92%20308%208337296>
>>>
>>> <http://venturedive.com/>
>>>
>>
>
> --
>
> Regards,
> *Sonam*
> Software Engineer
> Mobile: +92 3088337296
>
> <http://venturedive.com/>
>
Re: Random outputs for ARRAY_CONCAT_AGG fn zetasql
Posted by Sonam Ramchand <so...@venturedive.com>.
Yeah, that sounds right. But the only thing that confuses me is:
PAssert.that(stream).satisfies(row ->
assertThat("array_agg_concat_field", actual ,
containsInAnyOrder(Arrays.asList(1L,2L,3L,4L,5L,6L))));
How come I can access *actual* here when output is not materialized.
On Tue, Mar 2, 2021 at 10:49 PM Kyle Weaver <kc...@google.com> wrote:
> As you can see from existing tests, Beam doesn't materialize the output
> array directly. Instead you must use the PAssert API. I agree with Tyson's
> suggestion to use `satisfies`, which lets you do arbitrary assertions on
> the output data.
>
> On Tue, Mar 2, 2021 at 3:57 AM Sonam Ramchand <
> sonam.ramchand@venturedive.com> wrote:
>
>> Is there any way I can access the output array resulting from the sql
>> query? Then maybe I can sort and compare both *output array* and *expected
>> output array *for the test to pass.
>>
>> On Tue, Mar 2, 2021 at 12:24 AM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> Yea, the reason is that SQL relations are not ordered. So any ordering
>>> of [1, 2, 3, 4] and [5, 6] and [7, 8, 9] is possible and correct.
>>>
>>> Kenn
>>>
>>> On Mon, Mar 1, 2021 at 11:01 AM Tyson Hamilton <ty...@google.com>
>>> wrote:
>>>
>>>> I didn't find anything like that after a brief look. What you could do
>>>> instead is something like:
>>>>
>>>> PAssert.thatSingleton(stream).satisfies( row -> assertThat("array_field
>>>> containsInAnyOrder", row.getArray("array_field"),
>>>> containsInAnyOrder(Arrays.asList(...)));
>>>>
>>>> using junit/hamcrest matchers. I didn't verify this works myself but it
>>>> should give you an idea for some next steps.
>>>>
>>>>
>>>> On Mon, Mar 1, 2021 at 12:37 AM Sonam Ramchand <
>>>> sonam.ramchand@venturedive.com> wrote:
>>>>
>>>>> Hi Devs,
>>>>> I have implemented the ARRAY_CONCAT_AGG function for zetasql dialect.
>>>>> I am trying to validate the test as:
>>>>>
>>>>> @Test
>>>>> public void testArrayConcatAggZetasql() {
>>>>> String sql =
>>>>> "SELECT ARRAY_CONCAT_AGG(x) AS array_concat_agg FROM (SELECT [1, 2, 3, 4] AS x UNION ALL SELECT [5, 6] UNION ALL SELECT [7, 8, 9])";
>>>>>
>>>>> ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
>>>>> BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
>>>>> PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
>>>>>
>>>>> Schema schema = Schema.builder().addArrayField("array_field", FieldType.INT64).build();
>>>>> PAssert.that(stream)
>>>>> .containsInAnyOrder(
>>>>> Row.withSchema(schema).addArray(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L).build());
>>>>> pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
>>>>> }
>>>>>
>>>>> Expected Output is: 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L.
>>>>> But I am getting randomly different outputs:
>>>>> 1. 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L
>>>>> 2. 5L, 6L, 7L, 8L, 9L, 1L, 2L, 3L, 4L
>>>>> 3. 7L, 8L, 9L, 5L, 6L, 1L, 2L, 3L, 4L
>>>>>
>>>>> As per my understanding, it is because of containsInAnyOrder function.
>>>>> Is there anything Like:
>>>>>
>>>>> PAssert.that(stream)
>>>>> .containsAnyOfThem(
>>>>> Row.withSchema(schema).addArray(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L).build(),
>>>>> Row.withSchema(schema).addArray(5L, 6L, 7L, 8L, 9L, 1L, 2L, 3L, 4L).build(),
>>>>> Row.withSchema(schema).addArray(7L, 8L, 9L, 5L, 6L, 1L, 2L, 3L, 4L).build());
>>>>>
>>>>> I would really appreciate if anyone can help me in knowing how to
>>>>> handle such scenario in Beam.
>>>>>
>>>>> Thanks!
>>>>> --
>>>>> Regards,
>>>>> *Sonam*
>>>>> Software Engineer
>>>>> Mobile: +92 3088337296 <+92%20308%208337296>
>>>>>
>>>>> <http://venturedive.com/>
>>>>>
>>>>
>>
>> --
>>
>> Regards,
>> *Sonam*
>> Software Engineer
>> Mobile: +92 3088337296 <+92%20308%208337296>
>>
>> <http://venturedive.com/>
>>
>
--
Regards,
*Sonam*
Software Engineer
Mobile: +92 3088337296
<http://venturedive.com/>
Re: Random outputs for ARRAY_CONCAT_AGG fn zetasql
Posted by Kyle Weaver <kc...@google.com>.
As you can see from existing tests, Beam doesn't materialize the output
array directly. Instead you must use the PAssert API. I agree with Tyson's
suggestion to use `satisfies`, which lets you do arbitrary assertions on
the output data.
On Tue, Mar 2, 2021 at 3:57 AM Sonam Ramchand <
sonam.ramchand@venturedive.com> wrote:
> Is there any way I can access the output array resulting from the sql
> query? Then maybe I can sort and compare both *output array* and *expected
> output array *for the test to pass.
>
> On Tue, Mar 2, 2021 at 12:24 AM Kenneth Knowles <ke...@apache.org> wrote:
>
>> Yea, the reason is that SQL relations are not ordered. So any ordering of
>> [1, 2, 3, 4] and [5, 6] and [7, 8, 9] is possible and correct.
>>
>> Kenn
>>
>> On Mon, Mar 1, 2021 at 11:01 AM Tyson Hamilton <ty...@google.com>
>> wrote:
>>
>>> I didn't find anything like that after a brief look. What you could do
>>> instead is something like:
>>>
>>> PAssert.thatSingleton(stream).satisfies( row -> assertThat("array_field
>>> containsInAnyOrder", row.getArray("array_field"),
>>> containsInAnyOrder(Arrays.asList(...)));
>>>
>>> using junit/hamcrest matchers. I didn't verify this works myself but it
>>> should give you an idea for some next steps.
>>>
>>>
>>> On Mon, Mar 1, 2021 at 12:37 AM Sonam Ramchand <
>>> sonam.ramchand@venturedive.com> wrote:
>>>
>>>> Hi Devs,
>>>> I have implemented the ARRAY_CONCAT_AGG function for zetasql dialect. I
>>>> am trying to validate the test as:
>>>>
>>>> @Test
>>>> public void testArrayConcatAggZetasql() {
>>>> String sql =
>>>> "SELECT ARRAY_CONCAT_AGG(x) AS array_concat_agg FROM (SELECT [1, 2, 3, 4] AS x UNION ALL SELECT [5, 6] UNION ALL SELECT [7, 8, 9])";
>>>>
>>>> ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
>>>> BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
>>>> PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
>>>>
>>>> Schema schema = Schema.builder().addArrayField("array_field", FieldType.INT64).build();
>>>> PAssert.that(stream)
>>>> .containsInAnyOrder(
>>>> Row.withSchema(schema).addArray(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L).build());
>>>> pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
>>>> }
>>>>
>>>> Expected Output is: 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L.
>>>> But I am getting randomly different outputs:
>>>> 1. 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L
>>>> 2. 5L, 6L, 7L, 8L, 9L, 1L, 2L, 3L, 4L
>>>> 3. 7L, 8L, 9L, 5L, 6L, 1L, 2L, 3L, 4L
>>>>
>>>> As per my understanding, it is because of containsInAnyOrder function.
>>>> Is there anything Like:
>>>>
>>>> PAssert.that(stream)
>>>> .containsAnyOfThem(
>>>> Row.withSchema(schema).addArray(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L).build(),
>>>> Row.withSchema(schema).addArray(5L, 6L, 7L, 8L, 9L, 1L, 2L, 3L, 4L).build(),
>>>> Row.withSchema(schema).addArray(7L, 8L, 9L, 5L, 6L, 1L, 2L, 3L, 4L).build());
>>>>
>>>> I would really appreciate if anyone can help me in knowing how to
>>>> handle such scenario in Beam.
>>>>
>>>> Thanks!
>>>> --
>>>> Regards,
>>>> *Sonam*
>>>> Software Engineer
>>>> Mobile: +92 3088337296 <+92%20308%208337296>
>>>>
>>>> <http://venturedive.com/>
>>>>
>>>
>
> --
>
> Regards,
> *Sonam*
> Software Engineer
> Mobile: +92 3088337296 <+92%20308%208337296>
>
> <http://venturedive.com/>
>
Re: Random outputs for ARRAY_CONCAT_AGG fn zetasql
Posted by Sonam Ramchand <so...@venturedive.com>.
Is there any way I can access the output array resulting from the sql
query? Then maybe I can sort and compare both *output array* and *expected
output array *for the test to pass.
On Tue, Mar 2, 2021 at 12:24 AM Kenneth Knowles <ke...@apache.org> wrote:
> Yea, the reason is that SQL relations are not ordered. So any ordering of
> [1, 2, 3, 4] and [5, 6] and [7, 8, 9] is possible and correct.
>
> Kenn
>
> On Mon, Mar 1, 2021 at 11:01 AM Tyson Hamilton <ty...@google.com> wrote:
>
>> I didn't find anything like that after a brief look. What you could do
>> instead is something like:
>>
>> PAssert.thatSingleton(stream).satisfies( row -> assertThat("array_field
>> containsInAnyOrder", row.getArray("array_field"),
>> containsInAnyOrder(Arrays.asList(...)));
>>
>> using junit/hamcrest matchers. I didn't verify this works myself but it
>> should give you an idea for some next steps.
>>
>>
>> On Mon, Mar 1, 2021 at 12:37 AM Sonam Ramchand <
>> sonam.ramchand@venturedive.com> wrote:
>>
>>> Hi Devs,
>>> I have implemented the ARRAY_CONCAT_AGG function for zetasql dialect. I
>>> am trying to validate the test as:
>>>
>>> @Test
>>> public void testArrayConcatAggZetasql() {
>>> String sql =
>>> "SELECT ARRAY_CONCAT_AGG(x) AS array_concat_agg FROM (SELECT [1, 2, 3, 4] AS x UNION ALL SELECT [5, 6] UNION ALL SELECT [7, 8, 9])";
>>>
>>> ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
>>> BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
>>> PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
>>>
>>> Schema schema = Schema.builder().addArrayField("array_field", FieldType.INT64).build();
>>> PAssert.that(stream)
>>> .containsInAnyOrder(
>>> Row.withSchema(schema).addArray(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L).build());
>>> pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
>>> }
>>>
>>> Expected Output is: 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L.
>>> But I am getting randomly different outputs:
>>> 1. 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L
>>> 2. 5L, 6L, 7L, 8L, 9L, 1L, 2L, 3L, 4L
>>> 3. 7L, 8L, 9L, 5L, 6L, 1L, 2L, 3L, 4L
>>>
>>> As per my understanding, it is because of containsInAnyOrder function.
>>> Is there anything Like:
>>>
>>> PAssert.that(stream)
>>> .containsAnyOfThem(
>>> Row.withSchema(schema).addArray(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L).build(),
>>> Row.withSchema(schema).addArray(5L, 6L, 7L, 8L, 9L, 1L, 2L, 3L, 4L).build(),
>>> Row.withSchema(schema).addArray(7L, 8L, 9L, 5L, 6L, 1L, 2L, 3L, 4L).build());
>>>
>>> I would really appreciate if anyone can help me in knowing how to handle
>>> such scenario in Beam.
>>>
>>> Thanks!
>>> --
>>> Regards,
>>> *Sonam*
>>> Software Engineer
>>> Mobile: +92 3088337296 <+92%20308%208337296>
>>>
>>> <http://venturedive.com/>
>>>
>>
--
Regards,
*Sonam*
Software Engineer
Mobile: +92 3088337296
<http://venturedive.com/>
Re: Random outputs for ARRAY_CONCAT_AGG fn zetasql
Posted by Kenneth Knowles <ke...@apache.org>.
Yea, the reason is that SQL relations are not ordered. So any ordering of
[1, 2, 3, 4] and [5, 6] and [7, 8, 9] is possible and correct.
Kenn
On Mon, Mar 1, 2021 at 11:01 AM Tyson Hamilton <ty...@google.com> wrote:
> I didn't find anything like that after a brief look. What you could do
> instead is something like:
>
> PAssert.thatSingleton(stream).satisfies( row -> assertThat("array_field
> containsInAnyOrder", row.getArray("array_field"),
> containsInAnyOrder(Arrays.asList(...)));
>
> using junit/hamcrest matchers. I didn't verify this works myself but it
> should give you an idea for some next steps.
>
>
> On Mon, Mar 1, 2021 at 12:37 AM Sonam Ramchand <
> sonam.ramchand@venturedive.com> wrote:
>
>> Hi Devs,
>> I have implemented the ARRAY_CONCAT_AGG function for zetasql dialect. I
>> am trying to validate the test as:
>>
>> @Test
>> public void testArrayConcatAggZetasql() {
>> String sql =
>> "SELECT ARRAY_CONCAT_AGG(x) AS array_concat_agg FROM (SELECT [1, 2, 3, 4] AS x UNION ALL SELECT [5, 6] UNION ALL SELECT [7, 8, 9])";
>>
>> ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
>> BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
>> PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
>>
>> Schema schema = Schema.builder().addArrayField("array_field", FieldType.INT64).build();
>> PAssert.that(stream)
>> .containsInAnyOrder(
>> Row.withSchema(schema).addArray(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L).build());
>> pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
>> }
>>
>> Expected Output is: 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L.
>> But I am getting randomly different outputs:
>> 1. 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L
>> 2. 5L, 6L, 7L, 8L, 9L, 1L, 2L, 3L, 4L
>> 3. 7L, 8L, 9L, 5L, 6L, 1L, 2L, 3L, 4L
>>
>> As per my understanding, it is because of containsInAnyOrder function. Is
>> there anything Like:
>>
>> PAssert.that(stream)
>> .containsAnyOfThem(
>> Row.withSchema(schema).addArray(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L).build(),
>> Row.withSchema(schema).addArray(5L, 6L, 7L, 8L, 9L, 1L, 2L, 3L, 4L).build(),
>> Row.withSchema(schema).addArray(7L, 8L, 9L, 5L, 6L, 1L, 2L, 3L, 4L).build());
>>
>> I would really appreciate if anyone can help me in knowing how to handle
>> such scenario in Beam.
>>
>> Thanks!
>> --
>> Regards,
>> *Sonam*
>> Software Engineer
>> Mobile: +92 3088337296 <+92%20308%208337296>
>>
>> <http://venturedive.com/>
>>
>
Re: Random outputs for ARRAY_CONCAT_AGG fn zetasql
Posted by Tyson Hamilton <ty...@google.com>.
I didn't find anything like that after a brief look. What you could do
instead is something like:
PAssert.thatSingleton(stream).satisfies( row -> assertThat("array_field
containsInAnyOrder", row.getArray("array_field"),
containsInAnyOrder(Arrays.asList(...)));
using junit/hamcrest matchers. I didn't verify this works myself but it
should give you an idea for some next steps.
On Mon, Mar 1, 2021 at 12:37 AM Sonam Ramchand <
sonam.ramchand@venturedive.com> wrote:
> Hi Devs,
> I have implemented the ARRAY_CONCAT_AGG function for zetasql dialect. I am
> trying to validate the test as:
>
> @Test
> public void testArrayConcatAggZetasql() {
> String sql =
> "SELECT ARRAY_CONCAT_AGG(x) AS array_concat_agg FROM (SELECT [1, 2, 3, 4] AS x UNION ALL SELECT [5, 6] UNION ALL SELECT [7, 8, 9])";
>
> ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
> BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
> PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
>
> Schema schema = Schema.builder().addArrayField("array_field", FieldType.INT64).build();
> PAssert.that(stream)
> .containsInAnyOrder(
> Row.withSchema(schema).addArray(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L).build());
> pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
> }
>
> Expected Output is: 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L.
> But I am getting randomly different outputs:
> 1. 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L
> 2. 5L, 6L, 7L, 8L, 9L, 1L, 2L, 3L, 4L
> 3. 7L, 8L, 9L, 5L, 6L, 1L, 2L, 3L, 4L
>
> As per my understanding, it is because of containsInAnyOrder function. Is
> there anything Like:
>
> PAssert.that(stream)
> .containsAnyOfThem(
> Row.withSchema(schema).addArray(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L).build(),
> Row.withSchema(schema).addArray(5L, 6L, 7L, 8L, 9L, 1L, 2L, 3L, 4L).build(),
> Row.withSchema(schema).addArray(7L, 8L, 9L, 5L, 6L, 1L, 2L, 3L, 4L).build());
>
> I would really appreciate if anyone can help me in knowing how to handle
> such scenario in Beam.
>
> Thanks!
> --
> Regards,
> *Sonam*
> Software Engineer
> Mobile: +92 3088337296 <+92%20308%208337296>
>
> <http://venturedive.com/>
>