[GitHub] [carbondata] xubo245 commented on a change in pull request #3188: [CARBONDATA-3336] Support configurable decode for loading binary data, support base64 and Hex decode.

[GitBox <gi...@apache.org>]

xubo245 commented on a change in pull request #3188: [CARBONDATA-3336] Support configurable decode for loading binary data, support base64 and Hex decode.
URL: https://github.com/apache/carbondata/pull/3188#discussion_r279174026
 
 

 ##########
 File path: processing/src/main/java/org/apache/carbondata/processing/loading/converter/impl/BinaryFieldConverterImpl.java
 ##########
 @@ -43,33 +47,55 @@
   private String nullformat;
   private boolean isEmptyBadRecord;
   private DataField dataField;
+  private String  binaryEncoder;
   public BinaryFieldConverterImpl(DataField dataField, String nullformat, int index,
-      boolean isEmptyBadRecord) {
+      boolean isEmptyBadRecord,String binaryEncoder) {
     this.dataType = dataField.getColumn().getDataType();
     this.dimension = (CarbonDimension) dataField.getColumn();
     this.nullformat = nullformat;
     this.index = index;
     this.isEmptyBadRecord = isEmptyBadRecord;
     this.dataField = dataField;
+    this.binaryEncoder = binaryEncoder;
   }
 
   @Override
   public void convert(CarbonRow row, BadRecordLogHolder logHolder)
       throws CarbonDataLoadingException {
-    if (row.getObject(index) instanceof String) {
-      row.update((((String) row.getObject(index)))
-          .getBytes(Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET)), index);
-    } else if (row.getObject(index) instanceof byte[]) {
-      row.update(row.getObject(index), index);
-    } else {
-      throw new CarbonDataLoadingException("Binary only support String and byte[] data type");
-    }
+    row.update(convert(row.getObject(index), logHolder), index);
   }
 
   @Override
   public Object convert(Object value, BadRecordLogHolder logHolder)
       throws RuntimeException {
-    return null;
+    if (value instanceof String) {
+      if (binaryEncoder.equalsIgnoreCase(
+          CarbonLoadOptionConstants.CARBON_OPTIONS_BINARY_DECODER_BASE64)) {
+        byte[] parsedValue = (String.valueOf(value))
+            .getBytes(Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
+        if (Base64.isArrayByteBase64(parsedValue)) {
+          parsedValue = Base64.decodeBase64(parsedValue);
+        } else {
+          //TODO: it should throw exception
 
 Review comment:
   Some committers require the binary should be the same with hive. but it will be different with hive if I add exception in here. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services