You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Shubham Chaurasia (Jira)" <ji...@apache.org> on 2020/02/03 14:06:00 UTC
[jira] [Commented] (SPARK-30714) DSV2: Vectorized datasource does
not have handling for ProlepticCalendar
[ https://issues.apache.org/jira/browse/SPARK-30714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028975#comment-17028975 ]
Shubham Chaurasia commented on SPARK-30714:
-------------------------------------------
Oh looks like https://issues.apache.org/jira/browse/SPARK-26651 handles this, will that be applicable for spark 2.3.x versions as well ?
> DSV2: Vectorized datasource does not have handling for ProlepticCalendar
> ------------------------------------------------------------------------
>
> Key: SPARK-30714
> URL: https://issues.apache.org/jira/browse/SPARK-30714
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.2
> Reporter: Shubham Chaurasia
> Priority: Major
>
> Consider the following scenarios -
> 1)
> {code:scala}
> scala> spark.read.format("com.shubham.MyDataSource").option("ts_millis", "1580736255261").load.show(false)
> MyDataSourceReader.createDataReaderFactories: ts_millis: 1580736255261
> Using LocalDateTime: 2020-02-03T13:24:15
> +-----------------------+
> |my_ts |
> +-----------------------+
> |2020-02-03 13:24:15.261|
> +-----------------------+
> {code}
> In above output, we can see that both the timestamps (the one logged and the one in dataframe) are equal(upto seconds)
> 2)
> {code:scala}
> scala> spark.read.format("com.shubham.MyDataSource").option("ts_millis", "-62135596800000").load.show(false)
> MyDataSourceReader.createDataReaderFactories: ts_millis: -62135596800000
> Using LocalDateTime: 0001-01-01T00:00
> +-------------------+
> |my_ts |
> +-------------------+
> |0001-01-03 00:00:00|
> +-------------------+
> {code}
> Here in 2), we can see that timestamp coming from DataReader is not converted properly according to proleptic calendar and hence the two timestamps are different.
> Code to Repro
> DataSourceReader
> {code:java}
> public class MyDataSourceReader implements DataSourceReader, SupportsScanColumnarBatch {
> private StructType schema;
> private long timestampToProduce = -62135596800000L;
> public MyDataSourceReader(Map<String, String> options) {
> initOptions(options);
> }
> private void initOptions(Map<String, String> options) {
> String ts = options.get("ts_millis");
> if (ts != null) {
> timestampToProduce = Long.parseLong(ts);
> }
> }
> @Override public StructType readSchema() {
> StructField[] fields = new StructField[1];
> fields[0] = new StructField("my_ts", DataTypes.TimestampType, true, Metadata.empty());
> schema = new StructType(fields);
> return schema;
> }
> @Override public List<DataReaderFactory<ColumnarBatch>> createBatchDataReaderFactories() {
> System.out.println("MyDataSourceReader.createDataReaderFactories: ts_millis: " + timestampToProduce);
> System.out.println("Using LocalDateTime: " + LocalDateTime.ofEpochSecond(timestampToProduce/1000, 0, ZoneOffset.UTC));
> List<DataReaderFactory<ColumnarBatch>> dataReaderFactories = new ArrayList<>();
> dataReaderFactories.add(new MyVectorizedTSProducerFactory(schema, timestampToProduce));
> return dataReaderFactories;
> }
> {code}
> DataReaderFactory & DataReader
> {code:java}
> public class MyVectorizedTSProducerFactory implements DataReaderFactory<ColumnarBatch> {
> private final StructType schema;
> private long timestampValueToProduce;
> public MyVectorizedTSProducerFactory(StructType schema, long timestampValueToProduce) {
> this.schema = schema;
> this.timestampValueToProduce = timestampValueToProduce;
> }
> @Override public DataReader<ColumnarBatch> createDataReader() {
> return new MyVectorizedProducerReader(schema, timestampValueToProduce);
> }
> public static class MyVectorizedProducerReader implements DataReader<ColumnarBatch> {
> private final StructType schema;
> private long timestampValueToProduce;
> private ColumnarBatch columnarBatch;
> // return just one batch for now
> private boolean batchRemaining = true;
> public MyVectorizedProducerReader(StructType schema, long timestampValueToProduce) {
> this.schema = schema;
> this.timestampValueToProduce = timestampValueToProduce;
> }
> @Override public boolean next() {
> return batchRemaining;
> }
> @Override public ColumnarBatch get() {
> batchRemaining = false;
> OnHeapColumnVector[] onHeapColumnVectors = OnHeapColumnVector.allocateColumns(1, schema);
> for (OnHeapColumnVector vector : onHeapColumnVectors) {
> // convert millis to micros
> vector.putLong(0, timestampValueToProduce * 1000);
> }
> columnarBatch = new ColumnarBatch(onHeapColumnVectors);
> columnarBatch.setNumRows(1);
> return columnarBatch;
> }
> @Override public void close() {
> if (columnarBatch != null) {
> columnarBatch.close();
> }
> }
> }
> }
> {code}
> Any workarounds/solutions for this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org