spark streaming insert data error

[Jocean shi <jo...@gmail.com>]

Hi:

I use spark streaming insert data into carbondata. I find that the schema
of my data don't match the schema of table.
after the method of class  RowParserImpl:

@Override
public Object[] parseRow(Object[] row) {
  if (row == null) {
    return new String[numberOfColumns];
  }
  // If number of columns are less in a row then create new array with
same size of header.
  if (row.length < numberOfColumns) {
    String[] temp = new String[numberOfColumns];
    System.arraycopy(row, 0, temp, 0, row.length);
    row = temp;
  }
  Object[] out = new Object[genericParsers.length];
  for (int i = 0; i < genericParsers.length; i++) {
    Object obj = row[inputMapping[i]];
    out[outputMapping[i]] = genericParsers[i].parse(obj);
  }
  return out;
}

The schema is "id,name,city,salary".but the table schema is
"name,city,id,salary" in

the class RowConverterImpl .

why? is bug?


Best

Jocean.shi

Re: 答复: spark streaming insert data error

[David CaiQiang <da...@gmail.com>]

There are all documents(include streaming table) under the following link.
https://github.com/apache/carbondata/tree/master/docs

You can find all examples in examples/spark2 module:
example 1 (support Update/Delete)
https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/StreamingUsingBatchLoadExample.scala

example 2 (not support Update/Delete)
https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/StructuredStreamingExample.scala
or:
https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/StreamingWithRowParserExample.scala



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

答复: spark streaming insert data error

[yanggs <ya...@guahao.com>]

I want to using spark streaming insert data into carbondata.
But I cann’t find any example for it.can you help for it. 
Another question is that Spark streaming is support update/delete carbondata table?

Re: spark streaming insert data error

[Jocean shi <jo...@gmail.com>]

HI yanggs:

The example is SparkStreamingExample in source code. Streaming table don't
support update/delete

Best
Jocean.shi

Jocean shi <jo...@gmail.com> 于2019年3月20日周三 下午4:09写道：

> HI:
>
> I dont't use CarbonTable.getCreateOrderColumn method.
> All the method controller is Carbondata.
> If my DataFrame schema order don't match table schema order then error
>
> Best
> Jocean.shi
>
>
> David CaiQiang <da...@gmail.com> 于2019年3月20日周三 下午3:48写道：
>
>> You can get table schema by CarbonTable.getCreateOrderColumn method.
>> It will return the correct table schema.
>>
>> "name,city,id,salary" is the order of column storage, it is not the table
>> schema.
>>
>>
>>
>> -----
>> Best Regards
>> David Cai
>> --
>> Sent from:
>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>>
>

Re: spark streaming insert data error

[Jocean shi <jo...@gmail.com>]

HI:

I dont't use CarbonTable.getCreateOrderColumn method.
All the method controller is Carbondata.
If my DataFrame schema order don't match table schema order then error

Best
Jocean.shi


David CaiQiang <da...@gmail.com> 于2019年3月20日周三 下午3:48写道：

> You can get table schema by CarbonTable.getCreateOrderColumn method.
> It will return the correct table schema.
>
> "name,city,id,salary" is the order of column storage, it is not the table
> schema.
>
>
>
> -----
> Best Regards
> David Cai
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>

Re: spark streaming insert data error

[David CaiQiang <da...@gmail.com>]

You can get table schema by CarbonTable.getCreateOrderColumn method.
It will return the correct table schema.

"name,city,id,salary" is the order of column storage, it is not the table
schema.



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/