You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "James Taylor (JIRA)" <ji...@apache.org> on 2015/11/12 21:35:10 UTC
[jira] [Comment Edited] (PHOENIX-2288) Phoenix-Spark: PDecimal
precision and scale aren't carried through to Spark DataFrame
[ https://issues.apache.org/jira/browse/PHOENIX-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15002876#comment-15002876 ]
James Taylor edited comment on PHOENIX-2288 at 11/12/15 8:34 PM:
-----------------------------------------------------------------
Thanks, [~jmahonin]. There's some very similar code in PhoenixRuntime (that doesn't match particular PDataType instances) for the toTypeString() implementation:
{code}
public static String getSqlTypeName(PColumn pCol) {
PDataType dataType = pCol.getDataType();
Integer maxLength = pCol.getMaxLength();
Integer scale = pCol.getScale();
return dataType.isArrayType() ? getArraySqlTypeName(maxLength, scale, dataType) : appendMaxLengthAndScale(maxLength, scale, dataType.getSqlTypeName());
}
public static String getArraySqlTypeName(@Nullable Integer maxLength, @Nullable Integer scale, PDataType arrayType) {
String baseTypeSqlName = PDataType.arrayBaseType(arrayType).getSqlTypeName();
return appendMaxLengthAndScale(maxLength, scale, baseTypeSqlName) + " " + ARRAY_TYPE_SUFFIX; // for ex - decimal(10,2) ARRAY
}
private static String appendMaxLengthAndScale(@Nullable Integer maxLength, @Nullable Integer scale, String sqlTypeName) {
if (maxLength != null) {
sqlTypeName = sqlTypeName + "(" + maxLength;
if (scale != null) {
sqlTypeName = sqlTypeName + "," + scale; // has both max length and scale. For ex- decimal(10,2)
}
sqlTypeName = sqlTypeName + ")";
}
return sqlTypeName;
}
{code}
Would you mind refactoring that as necessary and perhaps putting it in SchemaUtil?
Everything else looks great!
was (Author: jamestaylor):
Thanks, [~jmahonin]. There's some very similar code in PhoenixRuntime (that doesn't match particular PDataType instances) for the toTypeString() implementation:
{code}
public static String getSqlTypeName(PColumn pCol) {
PDataType dataType = pCol.getDataType();
Integer maxLength = pCol.getMaxLength();
Integer scale = pCol.getScale();
return dataType.isArrayType() ? getArraySqlTypeName(maxLength, scale, dataType) : appendMaxLengthAndScale(maxLength, scale, dataType.getSqlTypeName());
}
public static String getArraySqlTypeName(@Nullable Integer maxLength, @Nullable Integer scale, PDataType arrayType) {
String baseTypeSqlName = PDataType.arrayBaseType(arrayType).getSqlTypeName();
return appendMaxLengthAndScale(maxLength, scale, baseTypeSqlName) + " " + ARRAY_TYPE_SUFFIX; // for ex - decimal(10,2) ARRAY
}
private static String appendMaxLengthAndScale(@Nullable Integer maxLength, @Nullable Integer scale, String sqlTypeName) {
if (maxLength != null) {
sqlTypeName = sqlTypeName + "(" + maxLength;
if (scale != null) {
sqlTypeName = sqlTypeName + "," + scale; // has both max length and scale. For ex- decimal(10,2)
}
sqlTypeName = sqlTypeName + ")";
}
return sqlTypeName;
}
{code}
Would you mind refactoring that as necessary and perhaps putting it in SchemaUtil?
> Phoenix-Spark: PDecimal precision and scale aren't carried through to Spark DataFrame
> -------------------------------------------------------------------------------------
>
> Key: PHOENIX-2288
> URL: https://issues.apache.org/jira/browse/PHOENIX-2288
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.5.2
> Reporter: Josh Mahonin
> Attachments: PHOENIX-2288-v2.patch, PHOENIX-2288.patch
>
>
> When loading a Spark dataframe from a Phoenix table with a 'DECIMAL' type, the underlying precision and scale aren't carried forward to Spark.
> The Spark catalyst schema converter should load these from the underlying column. These appear to be exposed in the ResultSetMetaData, but if there was a way to expose these somehow through ColumnInfo, it would be cleaner.
> I'm not sure if Pig has the same issues or not, but I suspect it may.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)