You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2022/09/21 17:35:16 UTC
[doris] 07/08: [Enhancement](sparkload) cast the src slot types of bitmap columns to `bitmap` when FE push tasks in spark load (#12394)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 42f11ab110deb68a5677a91ac7589280add31340
Author: spaces-x <we...@gmail.com>
AuthorDate: Wed Sep 7 10:07:38 2022 +0800
[Enhancement](sparkload) cast the src slot types of bitmap columns to `bitmap` when FE push tasks in spark load (#12394)
In the current spark load implementation, the types of source data, that BE reads from the Broker, are all set to varchar.
However, the two types of varchar and bitmap are not compatible anymore after version 1.1.0, which will cause spark load failure.
An example of spark load error message:
detailMessage = type not match, originType=VARCHAR(*), targeType=BITMAP
Describe your changes.
Set the src type of the bitmap columns from varchar to bitmapwhen fe pushtasks.
---
.../main/java/org/apache/doris/load/loadv2/SparkLoadJob.java | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLoadJob.java b/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLoadJob.java
index 17c39b904a..751d2ff61a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLoadJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLoadJob.java
@@ -901,10 +901,18 @@ public class SparkLoadJob extends BulkLoadJob {
Map<String, SlotDescriptor> srcSlotDescByName = Maps.newHashMap();
for (Column column : columns) {
SlotDescriptor srcSlotDesc = descTable.addSlotDescriptor(srcTupleDesc);
- srcSlotDesc.setType(ScalarType.createType(PrimitiveType.VARCHAR));
srcSlotDesc.setIsMaterialized(true);
srcSlotDesc.setIsNullable(true);
- srcSlotDesc.setColumn(new Column(column.getName(), PrimitiveType.VARCHAR));
+
+ if (column.getDataType() == PrimitiveType.BITMAP) {
+ // cast to bitmap when the target column type is bitmap
+ srcSlotDesc.setType(ScalarType.createType(PrimitiveType.BITMAP));
+ srcSlotDesc.setColumn(new Column(column.getName(), PrimitiveType.BITMAP));
+ } else {
+ srcSlotDesc.setType(ScalarType.createType(PrimitiveType.VARCHAR));
+ srcSlotDesc.setColumn(new Column(column.getName(), PrimitiveType.VARCHAR));
+ }
+
params.addToSrcSlotIds(srcSlotDesc.getId().asInt());
srcSlotDescByName.put(column.getName(), srcSlotDesc);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org