You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by "xubo245 (JIRA)" <ji...@apache.org> on 2018/11/20 02:29:00 UTC
[jira] [Updated] (CARBONDATA-2999) support read schema from S3
[ https://issues.apache.org/jira/browse/CARBONDATA-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
xubo245 updated CARBONDATA-2999:
--------------------------------
Description:
support read schema from S3
Code:
{code:java}
String path = "s3a://sdk/WriterOutput/carbondata5";
if (args.length > 3) {
path=args[3];
}
Schema schema = CarbonSchemaReader.readSchema(path);
System.out.println(schema.getFieldsLength());
{code}
Exception:
{code:java}
WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile.<init>(AbstractDFSCarbonFile.java:74)
at org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile.<init>(AbstractDFSCarbonFile.java:66)
at org.apache.carbondata.core.datastore.filesystem.HDFSCarbonFile.<init>(HDFSCarbonFile.java:41)
at org.apache.carbondata.core.datastore.filesystem.S3CarbonFile.<init>(S3CarbonFile.java:41)
at org.apache.carbondata.core.datastore.impl.DefaultFileTypeProvider.getCarbonFile(DefaultFileTypeProvider.java:53)
at org.apache.carbondata.core.datastore.impl.FileFactory.getCarbonFile(FileFactory.java:99)
at org.apache.carbondata.sdk.file.CarbonSchemaReader.getCarbonFile(CarbonSchemaReader.java:79)
at org.apache.carbondata.sdk.file.CarbonSchemaReader.readSchema(CarbonSchemaReader.java:150)
at org.apache.carbondata.sdk.file.CarbonSchemaReader.readSchema(CarbonSchemaReader.java:109)
at org.apache.carbondata.examples.sdk.SDKS3SchemaReadExample.main(SDKS3SchemaReadExample.java:51)
{code}
was:support read schema from S3
> support read schema from S3
> ---------------------------
>
> Key: CARBONDATA-2999
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2999
> Project: CarbonData
> Issue Type: Sub-task
> Affects Versions: 1.5.0
> Reporter: xubo245
> Assignee: xubo245
> Priority: Major
>
> support read schema from S3
> Code:
> {code:java}
> String path = "s3a://sdk/WriterOutput/carbondata5";
> if (args.length > 3) {
> path=args[3];
> }
> Schema schema = CarbonSchemaReader.readSchema(path);
> System.out.println(schema.getFieldsLength());
> {code}
> Exception:
> {code:java}
> WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Exception in thread "main" com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
> at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
> at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
> at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
> at org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile.<init>(AbstractDFSCarbonFile.java:74)
> at org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile.<init>(AbstractDFSCarbonFile.java:66)
> at org.apache.carbondata.core.datastore.filesystem.HDFSCarbonFile.<init>(HDFSCarbonFile.java:41)
> at org.apache.carbondata.core.datastore.filesystem.S3CarbonFile.<init>(S3CarbonFile.java:41)
> at org.apache.carbondata.core.datastore.impl.DefaultFileTypeProvider.getCarbonFile(DefaultFileTypeProvider.java:53)
> at org.apache.carbondata.core.datastore.impl.FileFactory.getCarbonFile(FileFactory.java:99)
> at org.apache.carbondata.sdk.file.CarbonSchemaReader.getCarbonFile(CarbonSchemaReader.java:79)
> at org.apache.carbondata.sdk.file.CarbonSchemaReader.readSchema(CarbonSchemaReader.java:150)
> at org.apache.carbondata.sdk.file.CarbonSchemaReader.readSchema(CarbonSchemaReader.java:109)
> at org.apache.carbondata.examples.sdk.SDKS3SchemaReadExample.main(SDKS3SchemaReadExample.java:51)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)