You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@orc.apache.org by praveen reddy <pr...@gmail.com> on 2016/08/08 10:58:08 UTC

Timestamp and decimal datatypes

Hi,

i need to store Timestamp and Decimal datatype in ORC file. i declared them
as Date and BigDecimal in java but when i save them in ORC file, they are
storing as Struct type. i dont want to use Struct type.

can you please help on what data type i need to choose in Java. i am kind
of stuck on this.

Thanks,
Praveen

Re: Timestamp and decimal datatypes

Posted by praveen reddy <pr...@gmail.com>.

Thanks Owen, i will try it.

Thanks,
Praveen

On Mon, Aug 8, 2016 at 5:08 PM, Owen O'Malley <om...@apache.org> wrote:

> You can use the TypeDescription API to create the schema. Like:
>
>     TypeDescription schema = TypeDescription.createStruct()
>         .addField("time", TypeDescription.createTimestamp())
>         .addField("union", TypeDescription.createUnion()
>             .addUnionChild(TypeDescription.createInt())
>             .addUnionChild(TypeDescription.createString()))
>         .addField("decimal", TypeDescription.createDecimal()
>             .withPrecision(38)
>             .withScale(18));
>
> TypeDescription.fromString lets you be a little more compact:
>
>   TypeDescription schema = TypeDescription.fromString(
>     "struct<time:timestamp,uniontype<int,string>,decimal:
> decimal(38,18)>");
>
> .. Owen
>
> On Mon, Aug 8, 2016 at 2:01 PM, praveen reddy <praveen.onlinecourse@gmail.
> com> wrote:
>
>> i am on version 1.2. i am currently using ObjectInspectorFactory.getReflectionObjectInspector
>> to build the schema. will try to use DateWritable and HiveDecimalWritable.
>>
>> for using TypeDescription.fromString(), do we need to hardcode the
>> schema? if we need to hardcode i dont think that option will work for me. i
>> am looking into api further.
>>
>>
>> On Mon, Aug 8, 2016 at 3:14 PM, Owen O'Malley <om...@apache.org> wrote:
>>
>>> Which version are you using?
>>>
>>> If you are using Hive's ObjectInspectorFactory.getReflectionObjectInspector,
>>> you want to use DateWritable and HiveDecimalWritable.
>>>
>>> For even more control, you can create a TypeDescription and provide it
>>> via setSchema to the OrcFile.WriterOptions. In ORC 1.1 you can also use
>>> TypeDescription.fromString() to build TypeDescription from the type name.
>>>
>>> .. Owen
>>>
>>> On Mon, Aug 8, 2016 at 3:58 AM, praveen reddy <
>>> praveen.onlinecourse@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> i need to store Timestamp and Decimal datatype in ORC file. i declared
>>>> them as Date and BigDecimal in java but when i save them in ORC file, they
>>>> are storing as Struct type. i dont want to use Struct type.
>>>>
>>>> can you please help on what data type i need to choose in Java. i am
>>>> kind of stuck on this.
>>>>
>>>> Thanks,
>>>> Praveen
>>>>
>>>
>>>
>>
>

Re: Timestamp and decimal datatypes

Posted by Owen O'Malley <om...@apache.org>.

You can use the TypeDescription API to create the schema. Like:

    TypeDescription schema = TypeDescription.createStruct()
        .addField("time", TypeDescription.createTimestamp())
        .addField("union", TypeDescription.createUnion()
            .addUnionChild(TypeDescription.createInt())
            .addUnionChild(TypeDescription.createString()))
        .addField("decimal", TypeDescription.createDecimal()
            .withPrecision(38)
            .withScale(18));

TypeDescription.fromString lets you be a little more compact:

  TypeDescription schema = TypeDescription.fromString(
    "struct<time:timestamp,uniontype<int,string>,decimal:decimal(38,18)>");

.. Owen

On Mon, Aug 8, 2016 at 2:01 PM, praveen reddy <
praveen.onlinecourse@gmail.com> wrote:

> i am on version 1.2. i am currently using ObjectInspectorFactory.getReflectionObjectInspector
> to build the schema. will try to use DateWritable and HiveDecimalWritable.
>
> for using TypeDescription.fromString(), do we need to hardcode the schema?
> if we need to hardcode i dont think that option will work for me. i am
> looking into api further.
>
>
> On Mon, Aug 8, 2016 at 3:14 PM, Owen O'Malley <om...@apache.org> wrote:
>
>> Which version are you using?
>>
>> If you are using Hive's ObjectInspectorFactory.getReflectionObjectInspector,
>> you want to use DateWritable and HiveDecimalWritable.
>>
>> For even more control, you can create a TypeDescription and provide it
>> via setSchema to the OrcFile.WriterOptions. In ORC 1.1 you can also use
>> TypeDescription.fromString() to build TypeDescription from the type name.
>>
>> .. Owen
>>
>> On Mon, Aug 8, 2016 at 3:58 AM, praveen reddy <
>> praveen.onlinecourse@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> i need to store Timestamp and Decimal datatype in ORC file. i declared
>>> them as Date and BigDecimal in java but when i save them in ORC file, they
>>> are storing as Struct type. i dont want to use Struct type.
>>>
>>> can you please help on what data type i need to choose in Java. i am
>>> kind of stuck on this.
>>>
>>> Thanks,
>>> Praveen
>>>
>>
>>
>

Re: Timestamp and decimal datatypes

Posted by praveen reddy <pr...@gmail.com>.

i am on version 1.2. i am currently using
ObjectInspectorFactory.getReflectionObjectInspector to build the schema.
will try to use DateWritable and HiveDecimalWritable.

for using TypeDescription.fromString(), do we need to hardcode the schema?
if we need to hardcode i dont think that option will work for me. i am
looking into api further.

On Mon, Aug 8, 2016 at 3:14 PM, Owen O'Malley <om...@apache.org> wrote:

> Which version are you using?
>
> If you are using Hive's ObjectInspectorFactory.getReflectionObjectInspector,
> you want to use DateWritable and HiveDecimalWritable.
>
> For even more control, you can create a TypeDescription and provide it via
> setSchema to the OrcFile.WriterOptions. In ORC 1.1 you can also use
> TypeDescription.fromString() to build TypeDescription from the type name.
>
> .. Owen
>
> On Mon, Aug 8, 2016 at 3:58 AM, praveen reddy <praveen.onlinecourse@gmail.
> com> wrote:
>
>> Hi,
>>
>> i need to store Timestamp and Decimal datatype in ORC file. i declared
>> them as Date and BigDecimal in java but when i save them in ORC file, they
>> are storing as Struct type. i dont want to use Struct type.
>>
>> can you please help on what data type i need to choose in Java. i am kind
>> of stuck on this.
>>
>> Thanks,
>> Praveen
>>
>
>

Re: Timestamp and decimal datatypes

Posted by Owen O'Malley <om...@apache.org>.

Which version are you using?

If you are using
Hive's ObjectInspectorFactory.getReflectionObjectInspector, you want to use
DateWritable and HiveDecimalWritable.

For even more control, you can create a TypeDescription and provide it via
setSchema to the OrcFile.WriterOptions. In ORC 1.1 you can also use
TypeDescription.fromString() to build TypeDescription from the type name.

.. Owen

On Mon, Aug 8, 2016 at 3:58 AM, praveen reddy <
praveen.onlinecourse@gmail.com> wrote:

> Hi,
>
> i need to store Timestamp and Decimal datatype in ORC file. i declared
> them as Date and BigDecimal in java but when i save them in ORC file, they
> are storing as Struct type. i dont want to use Struct type.
>
> can you please help on what data type i need to choose in Java. i am kind
> of stuck on this.
>
> Thanks,
> Praveen
>