You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by xubo245 <gi...@git.apache.org> on 2018/04/12 09:37:34 UTC
[GitHub] carbondata pull request #2165: Add example to upload data to S3 by using SDK
GitHub user xubo245 opened a pull request:
https://github.com/apache/carbondata/pull/2165
Add example to upload data to S3 by using SDK
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xubo245/carbondata CARBONDATA-2338S3UsingSDK
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2165.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2165
----
commit 2a9b711a41c18f2c1a0d1432d9728b783531b4c1
Author: xubo245 <60...@...>
Date: 2018-04-12T09:12:38Z
[CARBONDATA-2338] Add example to upload data to S3 by using SDK
commit 788d3eb3fde73f2b62785e5099a86fc6e2a99820
Author: xubo245 <60...@...>
Date: 2018-04-12T09:36:17Z
optimize
----
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2165
LGTM
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2165
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5228/
---
[GitHub] carbondata pull request #2165: [CARBONDATA-2338][Test] Add example to upload...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2165
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2165
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3943/
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2165
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4970/
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2165
retest this please
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2165
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5100/
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2165
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4968/
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2165
@jackylk CI pass
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2165
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4420/
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2165
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5118/
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2165
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5260/
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2165
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3896/
---
[GitHub] carbondata pull request #2165: [CARBONDATA-2338][Test] Add example to upload...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2165#discussion_r181667504
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/S3UsingSDkExample.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.examples
+
+import org.apache.hadoop.fs.s3a.Constants.{ACCESS_KEY, ENDPOINT, SECRET_KEY}
+import org.apache.spark.sql.SparkSession
+import org.slf4j.{Logger, LoggerFactory}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.sdk.file.{CarbonWriter, Schema}
+
+object S3UsingSDKExample {
+
+ // prepare SDK writer output
+ def buildTestData(
+ path: String,
+ num: Int = 3,
+ persistSchema: Boolean = false): Any = {
+
+ // getCanonicalPath gives path with \, so code expects /. Need to handle in code ?
+ val writerPath = path.replace("\\", "/");
+
+ val schema = new StringBuilder()
--- End diff --
To make it simpler, can you use the object way to construct the schema object
---
[GitHub] carbondata pull request #2165: [CARBONDATA-2338][Test] Add example to upload...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2165#discussion_r181988052
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/S3UsingSDkExample.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.examples
+
+import org.apache.hadoop.fs.s3a.Constants.{ACCESS_KEY, ENDPOINT, SECRET_KEY}
+import org.apache.spark.sql.SparkSession
+import org.slf4j.{Logger, LoggerFactory}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.sdk.file.{CarbonWriter, Schema}
+
+object S3UsingSDKExample {
--- End diff --
ok, done
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2165
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4421/
---
[GitHub] carbondata pull request #2165: [CARBONDATA-2338][Test] Add example to upload...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2165#discussion_r183113401
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/S3UsingSDkExample.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.examples
+
+import org.apache.hadoop.fs.s3a.Constants.{ACCESS_KEY, ENDPOINT, SECRET_KEY}
+import org.apache.spark.sql.SparkSession
+import org.slf4j.{Logger, LoggerFactory}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.sdk.file.{CarbonWriter, Schema}
+
+object S3UsingSDKExample {
+
+ // prepare SDK writer output
+ def buildTestData(
+ path: String,
+ num: Int = 3,
+ persistSchema: Boolean = false): Any = {
+
+ // getCanonicalPath gives path with \, so code expects /. Need to handle in code ?
+ val writerPath = path.replace("\\", "/");
+
+ val schema = new StringBuilder()
--- End diff --
please refer to CSVCarbonWriterTest.testWriteFiles
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2165
retest this please
---
[GitHub] carbondata pull request #2165: [CARBONDATA-2338][Test] Add example to upload...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2165#discussion_r181667209
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/S3UsingSDkExample.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.examples
+
+import org.apache.hadoop.fs.s3a.Constants.{ACCESS_KEY, ENDPOINT, SECRET_KEY}
+import org.apache.spark.sql.SparkSession
+import org.slf4j.{Logger, LoggerFactory}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.sdk.file.{CarbonWriter, Schema}
+
+object S3UsingSDKExample {
--- End diff --
please add comment to describe this example
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2165
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3901/
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2165
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4149/
---
[GitHub] carbondata pull request #2165: [CARBONDATA-2338][Test] Add example to upload...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2165#discussion_r183291163
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/S3UsingSDkExample.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.examples
+
+import org.apache.hadoop.fs.s3a.Constants.{ACCESS_KEY, ENDPOINT, SECRET_KEY}
+import org.apache.spark.sql.SparkSession
+import org.slf4j.{Logger, LoggerFactory}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.sdk.file.{CarbonWriter, Schema}
+
+object S3UsingSDKExample {
+
+ // prepare SDK writer output
+ def buildTestData(
+ path: String,
+ num: Int = 3,
+ persistSchema: Boolean = false): Any = {
+
+ // getCanonicalPath gives path with \, so code expects /. Need to handle in code ?
+ val writerPath = path.replace("\\", "/");
+
+ val schema = new StringBuilder()
--- End diff --
ok,done
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2165
retest this please
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2165
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3897/
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2165
retest this please
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2165
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4079/
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2165
@ajantha-bhat @jackylk Please review it.
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2165
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3878/
---
[GitHub] carbondata pull request #2165: [CARBONDATA-2338][Test] Add example to upload...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2165#discussion_r182005295
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/S3UsingSDkExample.scala ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.examples
+
+import org.apache.hadoop.fs.s3a.Constants.{ACCESS_KEY, ENDPOINT, SECRET_KEY}
+import org.apache.spark.sql.SparkSession
+import org.slf4j.{Logger, LoggerFactory}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.sdk.file.{CarbonWriter, Schema}
+
+object S3UsingSDKExample {
+
+ // prepare SDK writer output
+ def buildTestData(
+ path: String,
+ num: Int = 3,
+ persistSchema: Boolean = false): Any = {
+
+ // getCanonicalPath gives path with \, so code expects /. Need to handle in code ?
+ val writerPath = path.replace("\\", "/");
+
+ val schema = new StringBuilder()
--- End diff --
I refer merged code in carbondata, there are many usage like that:
org.apache.carbondata.spark.testsuite.createTable.TestCarbonFileInputFormatWithExternalCarbonTable
org.apache.carbondata.spark.testsuite.createTable.TestUnmanagedCarbonTable
Keep it? or change all similar place in carbondata?
---
[GitHub] carbondata issue #2165: [CARBONDATA-2338][Test] Add example to upload data t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2165
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5122/
---