[GitHub] [arrow] youngfn opened a new issue, #14853: [C++][Streaming execution] can't write data after hash_distinct

[GitBox <gi...@apache.org>]

youngfn opened a new issue, #14853:
URL: https://github.com/apache/arrow/issues/14853

   ### Describe the usage question you have. Please include as many useful details as  possible.
   
   
   Hi, when I test the Streaming execution engine, I will always get an error like "Unsupported Type:list<item: int64>" in write node. This happened when I use hash_distinct in aggregate node (**but it will success with hash_count_distinct or hash_count**). 
   Is anything wrong with my demo? Is this a bug? I'm not sure. thx for any hint!
   ```
   //demo
   cp::Declaration::Sequence(	
   		{
   				{"scan", scan_node_options},
   				{"filter", cp::FilterNodeOptions{filter_opt}},
   				{"project", cp::ProjectNodeOptions{{
   						   cp::field_ref("id"), cp::field_ref("class_id"),cp::field_ref("gender"), cp::field_ref("age"),cp::field_ref("term"),
   						   expr}, {"id", "class_id", "name", "gender", "age", "score", "term"}}},
   				{"aggregate", cp::AggregateNodeOptions{/*aggregates=*/{{"hash_distinct", nullptr, "id", "distinct(id)"}}, {"id"}}},
   				{"write", write_node_options}
   		}
   ).AddToPlan(plan.get());
   
   if (!plan->Validate().ok()) {
   	std::cout << "plan is not validate" << std::endl;
   	return;
   }
   
   std::cout << "Execution Plan Created : " << plan->ToString() << std::endl;
   // // // start the ExecPlan
   plan->StartProducing();
   auto future = plan->finished();
   future.status();
   future.Wait();
   ```
   
   
   Error print:
   
   ```
   arrow error:Invalid: Unsupported Type:list<item: int64>
   arrow error:Invalid: Unsupported Type:list<item: int64>
   /tmp/tmp.GwaQRyi1BD/src/arrow/csv/writer.cc:454  MakePopulator(*schema->field(col), end_chars, options.delimiter, null_string, options.quoting_style, options.io_context.pool())
   arrow error:Invalid: Unsupported Type:list<item: int64>
   /tmp/tmp.GwaQRyi1BD/src/arrow/csv/writer.cc:454  MakePopulator(*schema->field(col), end_chars, options.delimiter, null_string, options.quoting_style, options.io_context.pool())
   /tmp/tmp.GwaQRyi1BD/src/arrow/dataset/file_csv.cc:335  csv::MakeCSVWriter(destination, schema, *csv_options->write_options)
   ```
   
   
   ### Component(s)
   
   C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] westonpace commented on issue #14853: [C++][Streaming execution] can't write data after hash_distinct

[GitBox <gi...@apache.org>]

westonpace commented on issue #14853:
URL: https://github.com/apache/arrow/issues/14853#issuecomment-1340620347

   The hash_count and hash_count_distinct functions return a single value for each group.  The hash_distinct function returns a list.  So, for example, given:
   
   key | value
   --- | ---
   1 | a
   1 | a
   1 | b
   2 | c
   3 | d
   
   You would get something like...
   
   key | count | count_distinct | distinct
   --- | --- | --- | ---
   1 | 3 | 2 | [a, b]
   2 | 1 | 1 | [c]
   3 | 1 | 1 | [d]
   
   The data type for column `distinct` is `list<int64>` and the CSV writer has no idea how to convert the above table into CSV (the CSV writer does not support `list` columns or `struct` columns or `map` columns).
   
   However, from your example, it looks like you might be after the distinct values of the key column.  If that is the case then you don't need any aggregates at all.
   
   ```
   {"aggregate", cp::AggregateNodeOptions{/*aggregates=*/{}, {"id"}}},
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] youngfn commented on issue #14853: [C++][Streaming execution] can't write data after hash_distinct

[GitBox <gi...@apache.org>]

youngfn commented on issue #14853:
URL: https://github.com/apache/arrow/issues/14853#issuecomment-1340682952

   Thank you so much.
   Indeed, I misunderstood the usage of hash_distinct(), I just try to get the distinct id. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] youngfn closed issue #14853: [C++][Streaming execution] can't write data after hash_distinct

[GitBox <gi...@apache.org>]

youngfn closed issue #14853: [C++][Streaming execution] can't write data after hash_distinct
URL: https://github.com/apache/arrow/issues/14853


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org