You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/04/20 19:51:05 UTC

[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5271: Remove hadoop dependency in Create Segment Command

Jackie-Jiang commented on a change in pull request #5271:
URL: https://github.com/apache/incubator-pinot/pull/5271#discussion_r411645241



##########
File path: pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/CreateSegmentCommand.java
##########
@@ -334,10 +333,10 @@ public void run() {
           for (int curr = 0; curr <= _retry; curr++) {
             try {
               SegmentGeneratorConfig config = new SegmentGeneratorConfig(segmentGeneratorConfig);
-
-              String localFile = dataFilePath.getName();
-              Path localFilePath = new Path(localFile);
-              dataDirPath.getFileSystem(new Configuration()).copyToLocalFile(dataFilePath, localFilePath);
+              URI dataFileUri = URI.create(dataFilePath);
+              String[] splits = dataFilePath.split("/");
+              String localFile = splits[splits.length - 1];

Review comment:
       Delete the local file after generating the segment? (Also recommend putting it into a temp dir)
   Also, not sure if it is a good idea directly using the file name because there might be conflict for nested dir.

##########
File path: pinot-tools/src/main/java/org/apache/pinot/tools/streams/githubevents/PullRequestMergedEventsStream.java
##########
@@ -206,41 +215,41 @@ public void start() {
    * Find commits, review comments, comments corresponding to this pull request event.
    * Construct a PullRequestMergedEvent with the help of the event, commits, review comments and comments.
    * Converts PullRequestMergedEvent to GenericRecord
+   * @param event
    */
-  private GenericRecord convertToPullRequestMergedGenericRecord(JsonElement eventJson)
+  private GenericRecord convertToPullRequestMergedGenericRecord(JsonNode event)
       throws IOException {
     GenericRecord genericRecord = null;
-    JsonObject event = eventJson.getAsJsonObject();
-    String type = event.get("type").getAsString();
+    String type = event.get("type").asText();
 
     if ("PullRequestEvent".equals(type)) {
-      JsonObject payload = event.get("payload").getAsJsonObject();
+      JsonNode payload = event.get("payload");
       if (payload != null) {
-        String action = payload.get("action").getAsString();
-        JsonObject pullRequest = payload.get("pull_request").getAsJsonObject();
-        String merged = pullRequest.get("merged").getAsString();
+        String action = payload.get("action").asText();
+        JsonNode pullRequest = payload.get("pull_request");
+        String merged = pullRequest.get("merged").asText();
         if ("closed".equals(action) && "true".equals(merged)) { // valid pull request merge event
 
-          JsonArray commits = null;
-          String commitsURL = pullRequest.get("commits_url").getAsString();
+          JsonNode commits = null;
+          String commitsURL = pullRequest.get("commits_url").asText();
           GitHubAPICaller.GitHubAPIResponse commitsResponse = _gitHubAPICaller.callAPI(commitsURL);
 
           if (commitsResponse.responseString != null) {
-            commits = new JsonParser().parse(commitsResponse.responseString).getAsJsonArray();
+            commits = OBJECT_MAPPER.reader().readTree(commitsResponse.responseString);

Review comment:
       ```suggestion
               commits = JsonUtils.stringToJsonNode(commitsResponse.responseString);
   ```
   Same for other places. No need to construct another object mapper




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org