You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2021/03/25 08:02:42 UTC

[GitHub] [drill] vdiravka opened a new pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

vdiravka opened a new pull request #2192:
URL: https://github.com/apache/drill/pull/2192


   # [DRILL-7828](https://issues.apache.org/jira/browse/DRILL-7828): Refactor Pcap and Pcapng format plugin
   
   ## Description
   
   It merges two (PCAP and PCAPNG) format plugins into one and placed it in the `contrib` module. Hence it uses two different readers for them. The decision, which reader to use is based on identifying PCAP file Magic Number
   
   ## Documentation
   An example of setting PCAP Format Plugin
   ```
           "pcap": {
             "type": "pcap",
             "extensions": ["pcap", "pcapng"],
             "stat" : false
           }
   ```
   
   ## Testing
   Added test case for the `pcapng` file, but `pcap` extension
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] vdiravka commented on a change in pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
vdiravka commented on a change in pull request #2192:
URL: https://github.com/apache/drill/pull/2192#discussion_r610068587



##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
##########
@@ -48,126 +49,61 @@
   private static final Logger logger = LoggerFactory.getLogger(PcapBatchReader.class);
 
   private FileSplit split;
-
   private PacketDecoder decoder;
-
   private InputStream fsStream;
-
   private RowSetLoader rowWriter;
-
   private int validBytes;
-
   private byte[] buffer;
-
   private int offset;
-
   private ScalarWriter typeWriter;
-
   private ScalarWriter timestampWriter;
-
   private ScalarWriter timestampMicroWriter;
-
   private ScalarWriter networkWriter;
-
   private ScalarWriter srcMacAddressWriter;
-
   private ScalarWriter dstMacAddressWriter;
-
   private ScalarWriter dstIPWriter;
-
   private ScalarWriter srcIPWriter;
-
   private ScalarWriter srcPortWriter;
-
   private ScalarWriter dstPortWriter;
-
   private ScalarWriter packetLengthWriter;
-
   private ScalarWriter tcpSessionWriter;
-
   private ScalarWriter tcpSequenceWriter;
-
   private ScalarWriter tcpAckNumberWriter;
-
   private ScalarWriter tcpFlagsWriter;
-
   private ScalarWriter tcpParsedFlagsWriter;
-
   private ScalarWriter tcpNsWriter;
-
   private ScalarWriter tcpCwrWriter;
-
   private ScalarWriter tcpEceWriter;
-
   private ScalarWriter tcpFlagsEceEcnCapableWriter;
-
   private ScalarWriter tcpFlagsCongestionWriter;
-
   private ScalarWriter tcpUrgWriter;
-
   private ScalarWriter tcpAckWriter;
-
   private ScalarWriter tcpPshWriter;
-
   private ScalarWriter tcpRstWriter;
-
   private ScalarWriter tcpSynWriter;
-
   private ScalarWriter tcpFinWriter;
-
   private ScalarWriter dataWriter;
-
   private ScalarWriter isCorruptWriter;
-
-  private final PcapReaderConfig readerConfig;
-
-
+  private final PcapFormatConfig readerConfig;
   // Writers for TCP Sessions
   private ScalarWriter sessionStartTimeWriter;
-
   private ScalarWriter sessionEndTimeWriter;
-
   private ScalarWriter sessionDurationWriter;
-
   private ScalarWriter connectionTimeWriter;
-
   private ScalarWriter packetCountWriter;
-
   private ScalarWriter originPacketCounterWriter;
-
   private ScalarWriter remotePacketCounterWriter;
-
   private ScalarWriter originDataVolumeWriter;
-
   private ScalarWriter remoteDataVolumeWriter;
-
   private ScalarWriter hostDataWriter;
-
   private ScalarWriter remoteDataWriter;
-
   private final int maxRecords;
-
   private Map<Long, TcpSession> sessionQueue;
 
 
-  public static class PcapReaderConfig {
-
-    protected final PcapFormatPlugin plugin;
-
-    public boolean sessionizeTCPStreams;
-
-    private final PcapFormatConfig config;
-
-    public PcapReaderConfig(PcapFormatPlugin plugin) {
-      this.plugin = plugin;
-      this.config = plugin.getConfig();
-      this.sessionizeTCPStreams = config.getSessionizeTCPStreams();
-    }
-  }
-
-  public PcapBatchReader(PcapReaderConfig readerConfig, int maxRecords) {
+  public PcapBatchReader(PcapFormatConfig readerConfig, int maxRecords) {
     this.readerConfig = readerConfig;
-    if (readerConfig.sessionizeTCPStreams) {
+    if (readerConfig.getSessionizeTCPStreams()) {

Review comment:
       I have added:
   ```
   SELECT type, packet_length, `timestamp` FROM table(dfs.`pcapng/sniff.pcapng` (type => 'pcapng', stat => false, sessionizeTCPStreams => true )) where type = 'ARP
   ```

##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/columns/ColumnsScanFramework.java
##########
@@ -103,7 +103,7 @@ protected void configure() {
   }
 
   @Override
-  protected SchemaNegotiatorImpl newNegotiator() {
+  public SchemaNegotiatorImpl newNegotiator() {

Review comment:
       Took this into account with new method:
   ```
       /**
        * @return FileScanFramework or empty object in case it is not binded yet with {@link #bind(ManagedScanFramework)}
        */
       protected Optional<FileScanFramework> fileFramework() {
         return Optional.ofNullable(fileFramework);
       }
   ```
   and
   ```
   It is not possible to detect file format, because the File Framework is not initialized. " +
                   "Trying to use default PCAP format plugin to read the file
   ```
   It will help for future new version of `FileReaderFactory`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] luocooong commented on pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
luocooong commented on pull request #2192:
URL: https://github.com/apache/drill/pull/2192#issuecomment-812849597


   @vdiravka Hi. Do we have any new revisions ready to submit? AFAIK, Now there is only an issue and related to the `conflict upragde`, Is right?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] vdiravka commented on a change in pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
vdiravka commented on a change in pull request #2192:
URL: https://github.com/apache/drill/pull/2192#discussion_r618839480



##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatConfig.java
##########
@@ -29,18 +29,23 @@
 import com.fasterxml.jackson.annotation.JsonProperty;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 
-@JsonTypeName(PcapngFormatConfig.NAME)
+@JsonTypeName(PcapFormatConfig.NAME)
 @JsonInclude(JsonInclude.Include.NON_DEFAULT)
-public class PcapngFormatConfig implements FormatPluginConfig {
+public class PcapFormatConfig implements FormatPluginConfig {
+  private static final List<String> DEFAULT_EXTNS = ImmutableList.of("pcap", "pcapng");
 
-  public static final String NAME = "pcapng";
+  public static final String NAME = "pcap";

Review comment:
       I did it :)
   Please check `TestPcapWithPersistentStore#pcapPluginBackwardCompatabilityTest()`. It is written similar to `TestPStoreProviders#zkBackwardCompatabilityTest()`.
   It serializes JSON file to `FileSystemConfig` object, put it to ZK PersistentStore and retrieves it from there as `PcapFormatConfig` and `PcapngFormatConfig` types.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] vdiravka commented on a change in pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
vdiravka commented on a change in pull request #2192:
URL: https://github.com/apache/drill/pull/2192#discussion_r610068587



##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
##########
@@ -48,126 +49,61 @@
   private static final Logger logger = LoggerFactory.getLogger(PcapBatchReader.class);
 
   private FileSplit split;
-
   private PacketDecoder decoder;
-
   private InputStream fsStream;
-
   private RowSetLoader rowWriter;
-
   private int validBytes;
-
   private byte[] buffer;
-
   private int offset;
-
   private ScalarWriter typeWriter;
-
   private ScalarWriter timestampWriter;
-
   private ScalarWriter timestampMicroWriter;
-
   private ScalarWriter networkWriter;
-
   private ScalarWriter srcMacAddressWriter;
-
   private ScalarWriter dstMacAddressWriter;
-
   private ScalarWriter dstIPWriter;
-
   private ScalarWriter srcIPWriter;
-
   private ScalarWriter srcPortWriter;
-
   private ScalarWriter dstPortWriter;
-
   private ScalarWriter packetLengthWriter;
-
   private ScalarWriter tcpSessionWriter;
-
   private ScalarWriter tcpSequenceWriter;
-
   private ScalarWriter tcpAckNumberWriter;
-
   private ScalarWriter tcpFlagsWriter;
-
   private ScalarWriter tcpParsedFlagsWriter;
-
   private ScalarWriter tcpNsWriter;
-
   private ScalarWriter tcpCwrWriter;
-
   private ScalarWriter tcpEceWriter;
-
   private ScalarWriter tcpFlagsEceEcnCapableWriter;
-
   private ScalarWriter tcpFlagsCongestionWriter;
-
   private ScalarWriter tcpUrgWriter;
-
   private ScalarWriter tcpAckWriter;
-
   private ScalarWriter tcpPshWriter;
-
   private ScalarWriter tcpRstWriter;
-
   private ScalarWriter tcpSynWriter;
-
   private ScalarWriter tcpFinWriter;
-
   private ScalarWriter dataWriter;
-
   private ScalarWriter isCorruptWriter;
-
-  private final PcapReaderConfig readerConfig;
-
-
+  private final PcapFormatConfig readerConfig;
   // Writers for TCP Sessions
   private ScalarWriter sessionStartTimeWriter;
-
   private ScalarWriter sessionEndTimeWriter;
-
   private ScalarWriter sessionDurationWriter;
-
   private ScalarWriter connectionTimeWriter;
-
   private ScalarWriter packetCountWriter;
-
   private ScalarWriter originPacketCounterWriter;
-
   private ScalarWriter remotePacketCounterWriter;
-
   private ScalarWriter originDataVolumeWriter;
-
   private ScalarWriter remoteDataVolumeWriter;
-
   private ScalarWriter hostDataWriter;
-
   private ScalarWriter remoteDataWriter;
-
   private final int maxRecords;
-
   private Map<Long, TcpSession> sessionQueue;
 
 
-  public static class PcapReaderConfig {
-
-    protected final PcapFormatPlugin plugin;
-
-    public boolean sessionizeTCPStreams;
-
-    private final PcapFormatConfig config;
-
-    public PcapReaderConfig(PcapFormatPlugin plugin) {
-      this.plugin = plugin;
-      this.config = plugin.getConfig();
-      this.sessionizeTCPStreams = config.getSessionizeTCPStreams();
-    }
-  }
-
-  public PcapBatchReader(PcapReaderConfig readerConfig, int maxRecords) {
+  public PcapBatchReader(PcapFormatConfig readerConfig, int maxRecords) {
     this.readerConfig = readerConfig;
-    if (readerConfig.sessionizeTCPStreams) {
+    if (readerConfig.getSessionizeTCPStreams()) {

Review comment:
       I have added test case in third commit:
   ```
   SELECT type, packet_length, `timestamp` FROM table(dfs.`pcapng/sniff.pcapng` (type => 'pcapng', stat => false, sessionizeTCPStreams => true )) where type = 'ARP
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] vdiravka commented on a change in pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
vdiravka commented on a change in pull request #2192:
URL: https://github.com/apache/drill/pull/2192#discussion_r610068587



##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
##########
@@ -48,126 +49,61 @@
   private static final Logger logger = LoggerFactory.getLogger(PcapBatchReader.class);
 
   private FileSplit split;
-
   private PacketDecoder decoder;
-
   private InputStream fsStream;
-
   private RowSetLoader rowWriter;
-
   private int validBytes;
-
   private byte[] buffer;
-
   private int offset;
-
   private ScalarWriter typeWriter;
-
   private ScalarWriter timestampWriter;
-
   private ScalarWriter timestampMicroWriter;
-
   private ScalarWriter networkWriter;
-
   private ScalarWriter srcMacAddressWriter;
-
   private ScalarWriter dstMacAddressWriter;
-
   private ScalarWriter dstIPWriter;
-
   private ScalarWriter srcIPWriter;
-
   private ScalarWriter srcPortWriter;
-
   private ScalarWriter dstPortWriter;
-
   private ScalarWriter packetLengthWriter;
-
   private ScalarWriter tcpSessionWriter;
-
   private ScalarWriter tcpSequenceWriter;
-
   private ScalarWriter tcpAckNumberWriter;
-
   private ScalarWriter tcpFlagsWriter;
-
   private ScalarWriter tcpParsedFlagsWriter;
-
   private ScalarWriter tcpNsWriter;
-
   private ScalarWriter tcpCwrWriter;
-
   private ScalarWriter tcpEceWriter;
-
   private ScalarWriter tcpFlagsEceEcnCapableWriter;
-
   private ScalarWriter tcpFlagsCongestionWriter;
-
   private ScalarWriter tcpUrgWriter;
-
   private ScalarWriter tcpAckWriter;
-
   private ScalarWriter tcpPshWriter;
-
   private ScalarWriter tcpRstWriter;
-
   private ScalarWriter tcpSynWriter;
-
   private ScalarWriter tcpFinWriter;
-
   private ScalarWriter dataWriter;
-
   private ScalarWriter isCorruptWriter;
-
-  private final PcapReaderConfig readerConfig;
-
-
+  private final PcapFormatConfig readerConfig;
   // Writers for TCP Sessions
   private ScalarWriter sessionStartTimeWriter;
-
   private ScalarWriter sessionEndTimeWriter;
-
   private ScalarWriter sessionDurationWriter;
-
   private ScalarWriter connectionTimeWriter;
-
   private ScalarWriter packetCountWriter;
-
   private ScalarWriter originPacketCounterWriter;
-
   private ScalarWriter remotePacketCounterWriter;
-
   private ScalarWriter originDataVolumeWriter;
-
   private ScalarWriter remoteDataVolumeWriter;
-
   private ScalarWriter hostDataWriter;
-
   private ScalarWriter remoteDataWriter;
-
   private final int maxRecords;
-
   private Map<Long, TcpSession> sessionQueue;
 
 
-  public static class PcapReaderConfig {
-
-    protected final PcapFormatPlugin plugin;
-
-    public boolean sessionizeTCPStreams;
-
-    private final PcapFormatConfig config;
-
-    public PcapReaderConfig(PcapFormatPlugin plugin) {
-      this.plugin = plugin;
-      this.config = plugin.getConfig();
-      this.sessionizeTCPStreams = config.getSessionizeTCPStreams();
-    }
-  }
-
-  public PcapBatchReader(PcapReaderConfig readerConfig, int maxRecords) {
+  public PcapBatchReader(PcapFormatConfig readerConfig, int maxRecords) {
     this.readerConfig = readerConfig;
-    if (readerConfig.sessionizeTCPStreams) {
+    if (readerConfig.getSessionizeTCPStreams()) {

Review comment:
       I'll add:
   ```
   SELECT type, packet_length, `timestamp` FROM table(dfs.`pcapng/sniff.pcapng` (type => 'pcapng', stat => false, sessionizeTCPStreams => true )) where type = 'ARP
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] vdiravka commented on pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
vdiravka commented on pull request #2192:
URL: https://github.com/apache/drill/pull/2192#issuecomment-812152774


   @paul-rogers I am familiar with this issue in Drill:
   https://stackoverflow.com/questions/50487014/apache-drill-1-12-0-failure-in-starting-embedded-drillbit-and-no-current-conn/50495562#50495562
   
   And I am absolutely agree about plugins configs versioning. We have it for parquet metdata, smth similar can be done for plugins. And other thing is proper migration tool for the new schema of storage plugins configs (every plugin configs change will require to update plugins configs version and migration on the new Drill sturtup with an old configs).
   
   But it sounds like a useful, but new feature. It shouldn't prevent to go further with current PR. Especially in current PR, I added code for an old plugins configs, so new Drill start-up will not cause any issues with an old plugin configs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] luocooong commented on pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
luocooong commented on pull request #2192:
URL: https://github.com/apache/drill/pull/2192#issuecomment-811878482


   > One other thought on the compatibility issue. We can perhaps fudge it here as we discussed. Perhaps it make sense to fix it correctly:
   > 
   > * Store a version number in the "persistent store" (ZK) each time we write to the store. This can't be a product version, it must be a persistent store format version. (Why? So you can change things in the SNAPSHOT release.)
   > * On each startup, check the persistent store version. If it is less than the current version, run an upgrade process. In particular, invoke a `versionxToy` conversion for each 'x' from the current store version to the current software version.
   > * In each conversion, look for changed configs and "do the right thing." For this PR, it would mean changing the plugin name and rewriting one of them ("pcap-ng" maybe) as needed. In some cases, it might mean deleting an unsupported plugin. Etc.
   > 
   > The wise thing would be to log the original JSON and any changes, so if things go wrong (and they will), there is a way to recover.
   
   Nice Idea. Versioning + Upgrade from previous + Recovery mode


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] luocooong commented on pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
luocooong commented on pull request #2192:
URL: https://github.com/apache/drill/pull/2192#issuecomment-812351104


   Actually, There is a JIRA (thanks @vdiravka) related to this ideas. [Storage Plugins Handler improvements](https://issues.apache.org/jira/browse/DRILL-6564)
   Well,  If this feature is implemented ASAP, it will be great. (1.19 or later ?)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] luocooong commented on pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
luocooong commented on pull request #2192:
URL: https://github.com/apache/drill/pull/2192#issuecomment-826403486


   @paul-rogers Thanks for your review and thanks for your time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] luocooong commented on pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
luocooong commented on pull request #2192:
URL: https://github.com/apache/drill/pull/2192#issuecomment-806697288


   @vdiravka Thanks for the nice submission. I'll do a review on this weekend.
   BTW, Has this PR implemented the `Streaming read for Pcap-NG format` feature?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] paul-rogers commented on a change in pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2192:
URL: https://github.com/apache/drill/pull/2192#discussion_r601836846



##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatConfig.java
##########
@@ -29,18 +29,23 @@
 import com.fasterxml.jackson.annotation.JsonProperty;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 
-@JsonTypeName(PcapngFormatConfig.NAME)
+@JsonTypeName(PcapFormatConfig.NAME)
 @JsonInclude(JsonInclude.Include.NON_DEFAULT)
-public class PcapngFormatConfig implements FormatPluginConfig {
+public class PcapFormatConfig implements FormatPluginConfig {
+  private static final List<String> DEFAULT_EXTNS = ImmutableList.of("pcap", "pcapng");
 
-  public static final String NAME = "pcapng";
+  public static final String NAME = "pcap";
   private final List<String> extensions;
   private final boolean stat;
+  private final boolean sessionizeTCPStreams;
 
   @JsonCreator
-  public PcapngFormatConfig(@JsonProperty("extensions") List<String> extensions, @JsonProperty("stat") boolean stat) {
-    this.extensions = extensions == null ? ImmutableList.of(PcapngFormatConfig.NAME) : ImmutableList.copyOf(extensions);
+  public PcapFormatConfig(@JsonProperty("extensions") List<String> extensions,
+                          @JsonProperty("stat") boolean stat,
+                          @JsonProperty("sessionizeTCPStreams") Boolean sessionizeTCPStreams) {

Review comment:
       People have to type this name. Brevity is good. Maybe just `sessionize` as the property name. Because this is PCAP, the only thing that *can* be sessionized are TCP streams.

##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatConfig.java
##########
@@ -15,7 +15,7 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.apache.drill.exec.store.pcapng;
+package org.apache.drill.exec.store.plugin;

Review comment:
       Seems odd for Pcap to be renamed to `plugin`. I would have thought that `plugin` would be more generic rather than one specific plugin. Suggestion: `pcap` as package name.

##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatConfig.java
##########
@@ -29,18 +29,23 @@
 import com.fasterxml.jackson.annotation.JsonProperty;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 
-@JsonTypeName(PcapngFormatConfig.NAME)
+@JsonTypeName(PcapFormatConfig.NAME)
 @JsonInclude(JsonInclude.Include.NON_DEFAULT)
-public class PcapngFormatConfig implements FormatPluginConfig {
+public class PcapFormatConfig implements FormatPluginConfig {
+  private static final List<String> DEFAULT_EXTNS = ImmutableList.of("pcap", "pcapng");
 
-  public static final String NAME = "pcapng";
+  public static final String NAME = "pcap";

Review comment:
       Note that changing this name may make existing systems unrecoverable. Drill cannot handle the case in which a plugin config stored in ZK changes names or structure (other than adding a field.) This change will requires all uses of PCAP to delete all plugins in ZK and start over. Is that the intent? Or, will there be code to read the old format and rename it to the new format (which must be done prior to Jackson deserialization)?
   
   Is there a unit test to check this? Creating one is a bit tricky: you have to simulate the old format in ZK, then read it with the new code.

##########
File path: contrib/format-pcapng/src/main/resources/bootstrap-format-plugins.json
##########
@@ -3,26 +3,29 @@
     "dfs": {
       "type": "file",
       "formats": {
-        "pcapng": {
-          "type": "pcapng",
+        "pcap": {
+          "type": "pcap",

Review comment:
       Breaking change; see above.

##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
##########
@@ -48,126 +49,61 @@
   private static final Logger logger = LoggerFactory.getLogger(PcapBatchReader.class);
 
   private FileSplit split;
-
   private PacketDecoder decoder;
-
   private InputStream fsStream;
-
   private RowSetLoader rowWriter;
-
   private int validBytes;
-
   private byte[] buffer;
-
   private int offset;
-
   private ScalarWriter typeWriter;
-
   private ScalarWriter timestampWriter;
-
   private ScalarWriter timestampMicroWriter;
-
   private ScalarWriter networkWriter;
-
   private ScalarWriter srcMacAddressWriter;
-
   private ScalarWriter dstMacAddressWriter;
-
   private ScalarWriter dstIPWriter;
-
   private ScalarWriter srcIPWriter;
-
   private ScalarWriter srcPortWriter;
-
   private ScalarWriter dstPortWriter;
-
   private ScalarWriter packetLengthWriter;
-
   private ScalarWriter tcpSessionWriter;
-
   private ScalarWriter tcpSequenceWriter;
-
   private ScalarWriter tcpAckNumberWriter;
-
   private ScalarWriter tcpFlagsWriter;
-
   private ScalarWriter tcpParsedFlagsWriter;
-
   private ScalarWriter tcpNsWriter;
-
   private ScalarWriter tcpCwrWriter;
-
   private ScalarWriter tcpEceWriter;
-
   private ScalarWriter tcpFlagsEceEcnCapableWriter;
-
   private ScalarWriter tcpFlagsCongestionWriter;
-
   private ScalarWriter tcpUrgWriter;
-
   private ScalarWriter tcpAckWriter;
-
   private ScalarWriter tcpPshWriter;
-
   private ScalarWriter tcpRstWriter;
-
   private ScalarWriter tcpSynWriter;
-
   private ScalarWriter tcpFinWriter;
-
   private ScalarWriter dataWriter;
-
   private ScalarWriter isCorruptWriter;
-
-  private final PcapReaderConfig readerConfig;
-
-
+  private final PcapFormatConfig readerConfig;
   // Writers for TCP Sessions
   private ScalarWriter sessionStartTimeWriter;
-
   private ScalarWriter sessionEndTimeWriter;
-
   private ScalarWriter sessionDurationWriter;
-
   private ScalarWriter connectionTimeWriter;
-
   private ScalarWriter packetCountWriter;
-
   private ScalarWriter originPacketCounterWriter;
-
   private ScalarWriter remotePacketCounterWriter;
-
   private ScalarWriter originDataVolumeWriter;
-
   private ScalarWriter remoteDataVolumeWriter;
-
   private ScalarWriter hostDataWriter;
-
   private ScalarWriter remoteDataWriter;
-
   private final int maxRecords;
-
   private Map<Long, TcpSession> sessionQueue;
 
 
-  public static class PcapReaderConfig {
-
-    protected final PcapFormatPlugin plugin;
-
-    public boolean sessionizeTCPStreams;
-
-    private final PcapFormatConfig config;
-
-    public PcapReaderConfig(PcapFormatPlugin plugin) {
-      this.plugin = plugin;
-      this.config = plugin.getConfig();
-      this.sessionizeTCPStreams = config.getSessionizeTCPStreams();
-    }
-  }
-
-  public PcapBatchReader(PcapReaderConfig readerConfig, int maxRecords) {
+  public PcapBatchReader(PcapFormatConfig readerConfig, int maxRecords) {
     this.readerConfig = readerConfig;
-    if (readerConfig.sessionizeTCPStreams) {
+    if (readerConfig.getSessionizeTCPStreams()) {

Review comment:
       Nit: no real need for a getter. The semantics of pugins require that the fields be public. A getter is needed only if we want to do something to that field.

##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/columns/ColumnsScanFramework.java
##########
@@ -103,7 +103,7 @@ protected void configure() {
   }
 
   @Override
-  protected SchemaNegotiatorImpl newNegotiator() {
+  public SchemaNegotiatorImpl newNegotiator() {

Review comment:
       Why is this now `public`? The `columns` framework was for CSV files using the `columns` column; this is an internal bit of structure not meant to be overridden.
   
   Also, would be better to invest time with the "new" version rather than this somewhat cumbersome older version.

##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatPlugin.java
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.plugin;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.scan.file.FileScanFramework;
+import org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileReaderFactory;
+import org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileScanBuilder;
+import org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.dfs.DrillFileSystem;
+import org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin;
+import org.apache.drill.exec.store.dfs.easy.EasySubScan;
+import org.apache.drill.exec.store.pcap.PcapBatchReader;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcapng.PcapngBatchReader;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.regex.Pattern;
+
+public class PcapFormatPlugin extends EasyFormatPlugin<PcapFormatConfig> {
+
+  static final Logger logger = LoggerFactory.getLogger(ManagedScanFramework.class);
+  private static PacketDecoder.FileFormat fileFormat = PacketDecoder.FileFormat.UNKNOWN;
+
+  public PcapFormatPlugin(String name,
+                          DrillbitContext context,
+                          Configuration fsConf,
+                          StoragePluginConfig storageConfig,
+                          PcapFormatConfig formatConfig) {
+    super(name, easyConfig(fsConf, formatConfig), context, storageConfig, formatConfig);
+  }
+
+  private static EasyFormatConfig easyConfig(Configuration fsConf, PcapFormatConfig pluginConfig) {
+    return EasyFormatConfig.builder()
+        .readable(true)
+        .writable(false)
+        .blockSplittable(false)
+        .compressible(true)
+        .extensions(pluginConfig.getExtensions())
+        .fsConf(fsConf)
+        .useEnhancedScan(true)
+        .supportsLimitPushdown(true)
+        .supportsProjectPushdown(true)
+        .defaultName(PcapFormatConfig.NAME)
+        .build();
+  }
+
+  private static class PcapReaderFactory extends FileReaderFactory {
+
+    private final PcapFormatConfig config;
+    private final EasySubScan scan;
+
+    public PcapReaderFactory(PcapFormatConfig config, EasySubScan scan) {
+      this.config = config;
+      this.scan = scan;
+    }
+
+    @Override
+    public ManagedReader<? extends FileSchemaNegotiator> newReader() {
+      fileFormat = fromMagicNumber(fileFramework);
+      return createReader(scan, config);
+    }
+  }
+
+  @Override
+  public ManagedReader<? extends FileSchemaNegotiator> newBatchReader(EasySubScan scan, OptionManager options) {
+    return createReader(scan, formatConfig);
+  }
+
+  private static ManagedReader<? extends FileSchemaNegotiator> createReader(EasySubScan scan, PcapFormatConfig config) {
+    switch(fileFormat) {
+      case PCAPNG: return new PcapngBatchReader(config, scan);
+      case PCAP:
+      case UNKNOWN:
+      default: return new PcapBatchReader(config, scan.getMaxRecords());
+    }
+  }
+
+  @Override
+  protected FileScanBuilder frameworkBuilder(OptionManager options, EasySubScan scan) {
+    FileScanBuilder builder = new FileScanBuilder();
+    builder.setReaderFactory(new PcapReaderFactory(formatConfig, scan));
+
+    initScanBuilder(builder, scan);
+    builder.nullType(Types.optional(MinorType.VARCHAR));
+    return builder;
+  }
+
+  /**
+   * Helper method to detect PCAP or PCAPNG file format based on file Magic Number
+   *
+   * @param fileFramework for obtaining InputStream
+   * @return PCAP/PCAPNG file format
+   */
+  private static PacketDecoder.FileFormat fromMagicNumber(FileScanFramework fileFramework) {
+    FileScanFramework.FileSchemaNegotiatorImpl negotiator = (FileScanFramework.FileSchemaNegotiatorImpl) fileFramework.newNegotiator();
+    DrillFileSystem dfs = negotiator.fileSystem();
+    Path path = dfs.makeQualified(negotiator.split().getPath());
+    try (InputStream inputStream = dfs.openPossiblyCompressedStream(path)) {
+      PacketDecoder decoder = new PacketDecoder(inputStream);
+      return decoder.getFileFormat();
+    } catch (IOException io) {
+      throw UserException
+              .dataReadError(io)
+              .addContext("File name:", path.toString())
+              .build(logger);
+    }
+  }
+
+  /**
+   * Detects PCAP or PCAPNG file format based on file extension. Not used now due to {@link #fromMagicNumber} usage for
+   * this purpose

Review comment:
       Drill conventions have been to remove unused code.

##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatPlugin.java
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.plugin;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.scan.file.FileScanFramework;
+import org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileReaderFactory;

Review comment:
       At some point it would be good to start using the "new" version of this stuff which is considerably simpler, and more forward looking, than this original version. For example, the "new" version provides better options for working with provided schemas.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] paul-rogers commented on a change in pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2192:
URL: https://github.com/apache/drill/pull/2192#discussion_r605396470



##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatConfig.java
##########
@@ -29,18 +29,23 @@
 import com.fasterxml.jackson.annotation.JsonProperty;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 
-@JsonTypeName(PcapngFormatConfig.NAME)
+@JsonTypeName(PcapFormatConfig.NAME)
 @JsonInclude(JsonInclude.Include.NON_DEFAULT)
-public class PcapngFormatConfig implements FormatPluginConfig {
+public class PcapFormatConfig implements FormatPluginConfig {
+  private static final List<String> DEFAULT_EXTNS = ImmutableList.of("pcap", "pcapng");
 
-  public static final String NAME = "pcapng";
+  public static final String NAME = "pcap";

Review comment:
       There may be a trick to test this. In some of the storage plugin rework we did, we added some tests to push things into the persistent store. You might be able to look at that and write an old-style config to the store as JSON, then try to retrieve it in a query.

##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
##########
@@ -48,126 +49,61 @@
   private static final Logger logger = LoggerFactory.getLogger(PcapBatchReader.class);
 
   private FileSplit split;
-
   private PacketDecoder decoder;
-
   private InputStream fsStream;
-
   private RowSetLoader rowWriter;
-
   private int validBytes;
-
   private byte[] buffer;
-
   private int offset;
-
   private ScalarWriter typeWriter;
-
   private ScalarWriter timestampWriter;
-
   private ScalarWriter timestampMicroWriter;
-
   private ScalarWriter networkWriter;
-
   private ScalarWriter srcMacAddressWriter;
-
   private ScalarWriter dstMacAddressWriter;
-
   private ScalarWriter dstIPWriter;
-
   private ScalarWriter srcIPWriter;
-
   private ScalarWriter srcPortWriter;
-
   private ScalarWriter dstPortWriter;
-
   private ScalarWriter packetLengthWriter;
-
   private ScalarWriter tcpSessionWriter;
-
   private ScalarWriter tcpSequenceWriter;
-
   private ScalarWriter tcpAckNumberWriter;
-
   private ScalarWriter tcpFlagsWriter;
-
   private ScalarWriter tcpParsedFlagsWriter;
-
   private ScalarWriter tcpNsWriter;
-
   private ScalarWriter tcpCwrWriter;
-
   private ScalarWriter tcpEceWriter;
-
   private ScalarWriter tcpFlagsEceEcnCapableWriter;
-
   private ScalarWriter tcpFlagsCongestionWriter;
-
   private ScalarWriter tcpUrgWriter;
-
   private ScalarWriter tcpAckWriter;
-
   private ScalarWriter tcpPshWriter;
-
   private ScalarWriter tcpRstWriter;
-
   private ScalarWriter tcpSynWriter;
-
   private ScalarWriter tcpFinWriter;
-
   private ScalarWriter dataWriter;
-
   private ScalarWriter isCorruptWriter;
-
-  private final PcapReaderConfig readerConfig;
-
-
+  private final PcapFormatConfig readerConfig;
   // Writers for TCP Sessions
   private ScalarWriter sessionStartTimeWriter;
-
   private ScalarWriter sessionEndTimeWriter;
-
   private ScalarWriter sessionDurationWriter;
-
   private ScalarWriter connectionTimeWriter;
-
   private ScalarWriter packetCountWriter;
-
   private ScalarWriter originPacketCounterWriter;
-
   private ScalarWriter remotePacketCounterWriter;
-
   private ScalarWriter originDataVolumeWriter;
-
   private ScalarWriter remoteDataVolumeWriter;
-
   private ScalarWriter hostDataWriter;
-
   private ScalarWriter remoteDataWriter;
-
   private final int maxRecords;
-
   private Map<Long, TcpSession> sessionQueue;
 
 
-  public static class PcapReaderConfig {
-
-    protected final PcapFormatPlugin plugin;
-
-    public boolean sessionizeTCPStreams;
-
-    private final PcapFormatConfig config;
-
-    public PcapReaderConfig(PcapFormatPlugin plugin) {
-      this.plugin = plugin;
-      this.config = plugin.getConfig();
-      this.sessionizeTCPStreams = config.getSessionizeTCPStreams();
-    }
-  }
-
-  public PcapBatchReader(PcapReaderConfig readerConfig, int maxRecords) {
+  public PcapBatchReader(PcapFormatConfig readerConfig, int maxRecords) {
     this.readerConfig = readerConfig;
-    if (readerConfig.sessionizeTCPStreams) {
+    if (readerConfig.getSessionizeTCPStreams()) {

Review comment:
       Good question. I think the issue is tied up with table functions. Try using your `private` members in a query that uses table functions to set plugin properties. This is a bit hazy now, but we changed something about that a while back to make things more general.
   
   In fact, a good rule of thumb would be to always add a table function unit test for each plugin and its properties.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] paul-rogers commented on pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on pull request #2192:
URL: https://github.com/apache/drill/pull/2192#issuecomment-811676260


   One other thought on the compatibility issue. We can perhaps fudge it here as we discussed. Perhaps it make sense to fix it correctly:
   
   * Store a version number in the "persistent store" (ZK) each time we write to the store. This can't be a product version, it must be a persistent store format version. (Why? So you can change things in the SNAPSHOT release.)
   * On each startup, check the persistent store version. If it is less than the current version, run an upgrade process. In particular, invoke a `versionxToy` conversion for each 'x' from the current store version to the current software version.
   * In each conversion, look for changed configs and "do the right thing." For this PR, it would mean changing the plugin name and rewriting one of them ("pcap-ng" maybe) as needed. In some cases, it might mean deleting an unsupported plugin. Etc.
   
   The wise thing would be to log the original JSON and any changes, so if things go wrong (and they will), there is a way to recover.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] vdiravka commented on a change in pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
vdiravka commented on a change in pull request #2192:
URL: https://github.com/apache/drill/pull/2192#discussion_r603043283



##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/columns/ColumnsScanFramework.java
##########
@@ -103,7 +103,7 @@ protected void configure() {
   }
 
   @Override
-  protected SchemaNegotiatorImpl newNegotiator() {
+  public SchemaNegotiatorImpl newNegotiator() {

Review comment:
       I agree the negotiator is an internal staff of `fileFramework`, so not good to make it public.
   Changed it with adding `fileSystem()` method in `FileScanFramework`. For me it looks essential here.
   
   I think there is no need here to create something more powerful for just obtaining `DrillFileSystem` object.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] vdiravka commented on pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
vdiravka commented on pull request #2192:
URL: https://github.com/apache/drill/pull/2192#issuecomment-825296554


   As I told earlier we don't break backward compatibility: in second commit `Changes related to CR` I have added support for old plugins and in third commit `Changes related to CR2` I have added test case to check it. So we are fine to use the new config and don't fail, if old plugin config is places in PS
   @paul-rogers Could you pls check again?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] paul-rogers commented on a change in pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2192:
URL: https://github.com/apache/drill/pull/2192#discussion_r605397353



##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatPlugin.java
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.plugin;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.scan.file.FileScanFramework;
+import org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileReaderFactory;

Review comment:
       We don't have a ticket (that I filed). I got as far as doing the framework and updating CSV. My thought was, if we are refactoring a plugin, might as well use the newest stuff. But, it is fine to do that some other time if/when there is a compelling reason to do so.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] paul-rogers commented on a change in pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2192:
URL: https://github.com/apache/drill/pull/2192#discussion_r605398703



##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/columns/ColumnsScanFramework.java
##########
@@ -103,7 +103,7 @@ protected void configure() {
   }
 
   @Override
-  protected SchemaNegotiatorImpl newNegotiator() {
+  public SchemaNegotiatorImpl newNegotiator() {

Review comment:
       As it turns out, cleaning up this kind of issue is one of the things that the new version does.
   
   There is also a race condition. The schema negotiator is passed into the reader; it is not meant to be obtained elsewhere. In general, the framework does a bunch of initialization, getting things ready. Then, each reader is created, one at a time, passing in the schema negotiator. If we have code trying to access it at some other time, then we'd have to review all that code to see if new pattern works.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] paul-rogers commented on pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on pull request #2192:
URL: https://github.com/apache/drill/pull/2192#issuecomment-812924446


   Where are we on this one? We should avoid the "boil the ocean" temptation: the desire to fix too many things in one PR. If this PR will break upgrades, then the team can decide if that is OK for the handful of people who might be affected. Maybe document recovery in release notes, or be prepared to describe the steps on Slack or the mail lists.
   
   I am concerned about table functions. They are obscure, but are supposed to work. I'd like to see a test added so we can be sure that the non-public config variables do, in fact, work. Of course, you could argue that we can skip that until someone complains, then file another bug because, again, there are likely few users of this plugin and likely none of them use table functions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] luocooong commented on pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
luocooong commented on pull request #2192:
URL: https://github.com/apache/drill/pull/2192#issuecomment-826403486


   @paul-rogers Thanks for your review and thanks for your time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] vdiravka merged pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
vdiravka merged pull request #2192:
URL: https://github.com/apache/drill/pull/2192


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] vdiravka commented on a change in pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
vdiravka commented on a change in pull request #2192:
URL: https://github.com/apache/drill/pull/2192#discussion_r602931658



##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatConfig.java
##########
@@ -29,18 +29,23 @@
 import com.fasterxml.jackson.annotation.JsonProperty;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 
-@JsonTypeName(PcapngFormatConfig.NAME)
+@JsonTypeName(PcapFormatConfig.NAME)
 @JsonInclude(JsonInclude.Include.NON_DEFAULT)
-public class PcapngFormatConfig implements FormatPluginConfig {
+public class PcapFormatConfig implements FormatPluginConfig {
+  private static final List<String> DEFAULT_EXTNS = ImmutableList.of("pcap", "pcapng");
 
-  public static final String NAME = "pcapng";
+  public static final String NAME = "pcap";
   private final List<String> extensions;
   private final boolean stat;
+  private final boolean sessionizeTCPStreams;
 
   @JsonCreator
-  public PcapngFormatConfig(@JsonProperty("extensions") List<String> extensions, @JsonProperty("stat") boolean stat) {
-    this.extensions = extensions == null ? ImmutableList.of(PcapngFormatConfig.NAME) : ImmutableList.copyOf(extensions);
+  public PcapFormatConfig(@JsonProperty("extensions") List<String> extensions,
+                          @JsonProperty("stat") boolean stat,
+                          @JsonProperty("sessionizeTCPStreams") Boolean sessionizeTCPStreams) {

Review comment:
       Agree. But in was not introduced by me. This field was present in the original `pcap` format config:
   ```
     @JsonCreator
     public PcapFormatConfig(
         @JsonProperty("extensions") List<String> extensions,
         @JsonProperty("sessionizeTCPStreams") Boolean sessionizeTCPStreams) {
   ```
   It is need to left it for backward compatibility of old PCAP format plugin configs. Or I can create new one `sessionize` and if `sessionizeTCPStreams` is present in config, assign it to new `sessionize`. Sounds reasonable?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] vdiravka commented on a change in pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
vdiravka commented on a change in pull request #2192:
URL: https://github.com/apache/drill/pull/2192#discussion_r602925295



##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatConfig.java
##########
@@ -15,7 +15,7 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.apache.drill.exec.store.pcapng;
+package org.apache.drill.exec.store.plugin;

Review comment:
       It is occasionally. I intended to to add additional package `plugin`, to separate out classes related to `FormatPlugin` and `FormatConfig` from actual `pcap` and `pcapng` readers.
   So the actual package is `package org.apache.drill.exec.store.pcap.plugin;`

##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatConfig.java
##########
@@ -29,18 +29,23 @@
 import com.fasterxml.jackson.annotation.JsonProperty;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 
-@JsonTypeName(PcapngFormatConfig.NAME)
+@JsonTypeName(PcapFormatConfig.NAME)
 @JsonInclude(JsonInclude.Include.NON_DEFAULT)
-public class PcapngFormatConfig implements FormatPluginConfig {
+public class PcapFormatConfig implements FormatPluginConfig {
+  private static final List<String> DEFAULT_EXTNS = ImmutableList.of("pcap", "pcapng");
 
-  public static final String NAME = "pcapng";
+  public static final String NAME = "pcap";

Review comment:
       You are right. Backward compatibility wasn't preserved.
   I'll change it by adding two additional classes `PcapngFormatConfig`, `PcapngFormatPlugin` and mark them as deprecated, since they are needed only for older plugin configs.
   
   `UnitTest` for this is really tricky. Since json deserializing is possible only in the process of including formats during bootstrap. So the test cases with the following format plugin initialization are added:
   `cluster.defineFormat("dfs", "pcapng", new PcapFormatConfig(ImmutableList.of("pcapng"), true, false));`

##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatPlugin.java
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.plugin;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.scan.file.FileScanFramework;
+import org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileReaderFactory;
+import org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileScanBuilder;
+import org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.dfs.DrillFileSystem;
+import org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin;
+import org.apache.drill.exec.store.dfs.easy.EasySubScan;
+import org.apache.drill.exec.store.pcap.PcapBatchReader;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcapng.PcapngBatchReader;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.regex.Pattern;
+
+public class PcapFormatPlugin extends EasyFormatPlugin<PcapFormatConfig> {
+
+  static final Logger logger = LoggerFactory.getLogger(ManagedScanFramework.class);
+  private static PacketDecoder.FileFormat fileFormat = PacketDecoder.FileFormat.UNKNOWN;
+
+  public PcapFormatPlugin(String name,
+                          DrillbitContext context,
+                          Configuration fsConf,
+                          StoragePluginConfig storageConfig,
+                          PcapFormatConfig formatConfig) {
+    super(name, easyConfig(fsConf, formatConfig), context, storageConfig, formatConfig);
+  }
+
+  private static EasyFormatConfig easyConfig(Configuration fsConf, PcapFormatConfig pluginConfig) {
+    return EasyFormatConfig.builder()
+        .readable(true)
+        .writable(false)
+        .blockSplittable(false)
+        .compressible(true)
+        .extensions(pluginConfig.getExtensions())
+        .fsConf(fsConf)
+        .useEnhancedScan(true)
+        .supportsLimitPushdown(true)
+        .supportsProjectPushdown(true)
+        .defaultName(PcapFormatConfig.NAME)
+        .build();
+  }
+
+  private static class PcapReaderFactory extends FileReaderFactory {
+
+    private final PcapFormatConfig config;
+    private final EasySubScan scan;
+
+    public PcapReaderFactory(PcapFormatConfig config, EasySubScan scan) {
+      this.config = config;
+      this.scan = scan;
+    }
+
+    @Override
+    public ManagedReader<? extends FileSchemaNegotiator> newReader() {
+      fileFormat = fromMagicNumber(fileFramework);
+      return createReader(scan, config);
+    }
+  }
+
+  @Override
+  public ManagedReader<? extends FileSchemaNegotiator> newBatchReader(EasySubScan scan, OptionManager options) {
+    return createReader(scan, formatConfig);
+  }
+
+  private static ManagedReader<? extends FileSchemaNegotiator> createReader(EasySubScan scan, PcapFormatConfig config) {
+    switch(fileFormat) {
+      case PCAPNG: return new PcapngBatchReader(config, scan);
+      case PCAP:
+      case UNKNOWN:
+      default: return new PcapBatchReader(config, scan.getMaxRecords());
+    }
+  }
+
+  @Override
+  protected FileScanBuilder frameworkBuilder(OptionManager options, EasySubScan scan) {
+    FileScanBuilder builder = new FileScanBuilder();
+    builder.setReaderFactory(new PcapReaderFactory(formatConfig, scan));
+
+    initScanBuilder(builder, scan);
+    builder.nullType(Types.optional(MinorType.VARCHAR));
+    return builder;
+  }
+
+  /**
+   * Helper method to detect PCAP or PCAPNG file format based on file Magic Number
+   *
+   * @param fileFramework for obtaining InputStream
+   * @return PCAP/PCAPNG file format
+   */
+  private static PacketDecoder.FileFormat fromMagicNumber(FileScanFramework fileFramework) {
+    FileScanFramework.FileSchemaNegotiatorImpl negotiator = (FileScanFramework.FileSchemaNegotiatorImpl) fileFramework.newNegotiator();
+    DrillFileSystem dfs = negotiator.fileSystem();
+    Path path = dfs.makeQualified(negotiator.split().getPath());
+    try (InputStream inputStream = dfs.openPossiblyCompressedStream(path)) {
+      PacketDecoder decoder = new PacketDecoder(inputStream);
+      return decoder.getFileFormat();
+    } catch (IOException io) {
+      throw UserException
+              .dataReadError(io)
+              .addContext("File name:", path.toString())
+              .build(logger);
+    }
+  }
+
+  /**
+   * Detects PCAP or PCAPNG file format based on file extension. Not used now due to {@link #fromMagicNumber} usage for
+   * this purpose

Review comment:
       Agreed. Removed

##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatPlugin.java
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.plugin;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.scan.file.FileScanFramework;
+import org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileReaderFactory;

Review comment:
       It is used in almost all format plugins. So looks fine to refactor it in scope of "Storage plugins improvement" 
   @vvysotskyi Do we have a ticket for it?

##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/columns/ColumnsScanFramework.java
##########
@@ -103,7 +103,7 @@ protected void configure() {
   }
 
   @Override
-  protected SchemaNegotiatorImpl newNegotiator() {
+  public SchemaNegotiatorImpl newNegotiator() {

Review comment:
       I agree the negotiator is an internal staff of `fileFramework`, so not good to make it public.
   Change it with adding `fileSystem()` method in `FileScanFramework`. For me it looks essential here.
   
   I think there is no need here to create something more powerful for just obtaining `DrillFileSystem` object.

##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatConfig.java
##########
@@ -29,18 +29,23 @@
 import com.fasterxml.jackson.annotation.JsonProperty;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 
-@JsonTypeName(PcapngFormatConfig.NAME)
+@JsonTypeName(PcapFormatConfig.NAME)
 @JsonInclude(JsonInclude.Include.NON_DEFAULT)
-public class PcapngFormatConfig implements FormatPluginConfig {
+public class PcapFormatConfig implements FormatPluginConfig {
+  private static final List<String> DEFAULT_EXTNS = ImmutableList.of("pcap", "pcapng");
 
-  public static final String NAME = "pcapng";
+  public static final String NAME = "pcap";
   private final List<String> extensions;
   private final boolean stat;
+  private final boolean sessionizeTCPStreams;
 
   @JsonCreator
-  public PcapngFormatConfig(@JsonProperty("extensions") List<String> extensions, @JsonProperty("stat") boolean stat) {
-    this.extensions = extensions == null ? ImmutableList.of(PcapngFormatConfig.NAME) : ImmutableList.copyOf(extensions);
+  public PcapFormatConfig(@JsonProperty("extensions") List<String> extensions,
+                          @JsonProperty("stat") boolean stat,
+                          @JsonProperty("sessionizeTCPStreams") Boolean sessionizeTCPStreams) {

Review comment:
       Agree. But in was not introduced by me. This field was present in the original `pcap` format config:
   ```
     @JsonCreator
     public PcapFormatConfig(
         @JsonProperty("extensions") List<String> extensions,
         @JsonProperty("sessionizeTCPStreams") Boolean sessionizeTCPStreams) {
   ```

##########
File path: contrib/format-pcapng/src/main/resources/bootstrap-format-plugins.json
##########
@@ -3,26 +3,29 @@
     "dfs": {
       "type": "file",
       "formats": {
-        "pcapng": {
-          "type": "pcapng",
+        "pcap": {
+          "type": "pcap",

Review comment:
       resolved

##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
##########
@@ -48,126 +49,61 @@
   private static final Logger logger = LoggerFactory.getLogger(PcapBatchReader.class);
 
   private FileSplit split;
-
   private PacketDecoder decoder;
-
   private InputStream fsStream;
-
   private RowSetLoader rowWriter;
-
   private int validBytes;
-
   private byte[] buffer;
-
   private int offset;
-
   private ScalarWriter typeWriter;
-
   private ScalarWriter timestampWriter;
-
   private ScalarWriter timestampMicroWriter;
-
   private ScalarWriter networkWriter;
-
   private ScalarWriter srcMacAddressWriter;
-
   private ScalarWriter dstMacAddressWriter;
-
   private ScalarWriter dstIPWriter;
-
   private ScalarWriter srcIPWriter;
-
   private ScalarWriter srcPortWriter;
-
   private ScalarWriter dstPortWriter;
-
   private ScalarWriter packetLengthWriter;
-
   private ScalarWriter tcpSessionWriter;
-
   private ScalarWriter tcpSequenceWriter;
-
   private ScalarWriter tcpAckNumberWriter;
-
   private ScalarWriter tcpFlagsWriter;
-
   private ScalarWriter tcpParsedFlagsWriter;
-
   private ScalarWriter tcpNsWriter;
-
   private ScalarWriter tcpCwrWriter;
-
   private ScalarWriter tcpEceWriter;
-
   private ScalarWriter tcpFlagsEceEcnCapableWriter;
-
   private ScalarWriter tcpFlagsCongestionWriter;
-
   private ScalarWriter tcpUrgWriter;
-
   private ScalarWriter tcpAckWriter;
-
   private ScalarWriter tcpPshWriter;
-
   private ScalarWriter tcpRstWriter;
-
   private ScalarWriter tcpSynWriter;
-
   private ScalarWriter tcpFinWriter;
-
   private ScalarWriter dataWriter;
-
   private ScalarWriter isCorruptWriter;
-
-  private final PcapReaderConfig readerConfig;
-
-
+  private final PcapFormatConfig readerConfig;
   // Writers for TCP Sessions
   private ScalarWriter sessionStartTimeWriter;
-
   private ScalarWriter sessionEndTimeWriter;
-
   private ScalarWriter sessionDurationWriter;
-
   private ScalarWriter connectionTimeWriter;
-
   private ScalarWriter packetCountWriter;
-
   private ScalarWriter originPacketCounterWriter;
-
   private ScalarWriter remotePacketCounterWriter;
-
   private ScalarWriter originDataVolumeWriter;
-
   private ScalarWriter remoteDataVolumeWriter;
-
   private ScalarWriter hostDataWriter;
-
   private ScalarWriter remoteDataWriter;
-
   private final int maxRecords;
-
   private Map<Long, TcpSession> sessionQueue;
 
 
-  public static class PcapReaderConfig {
-
-    protected final PcapFormatPlugin plugin;
-
-    public boolean sessionizeTCPStreams;
-
-    private final PcapFormatConfig config;
-
-    public PcapReaderConfig(PcapFormatPlugin plugin) {
-      this.plugin = plugin;
-      this.config = plugin.getConfig();
-      this.sessionizeTCPStreams = config.getSessionizeTCPStreams();
-    }
-  }
-
-  public PcapBatchReader(PcapReaderConfig readerConfig, int maxRecords) {
+  public PcapBatchReader(PcapFormatConfig readerConfig, int maxRecords) {
     this.readerConfig = readerConfig;
-    if (readerConfig.sessionizeTCPStreams) {
+    if (readerConfig.getSessionizeTCPStreams()) {

Review comment:
       It makes sense. It means it is needed to make these fields to be `public`. But such fields of other plugins are also private. Does that mean they should be changed to public eventually too?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] luocooong commented on pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
luocooong commented on pull request #2192:
URL: https://github.com/apache/drill/pull/2192#issuecomment-812963481


   @dzamo Hello. Is it possible to add a docs (when the [DRILL-6564](https://issues.apache.org/jira/browse/DRILL-6564) or related issues not resolved) that describe the troubleshooting about `conflict upgrade` at the phase of upgrade (from old plugin config). thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] paul-rogers commented on a change in pull request #2192: DRILL-7828: Refactor Pcap and Pcapng format plugin

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2192:
URL: https://github.com/apache/drill/pull/2192#discussion_r605396832



##########
File path: contrib/format-pcapng/src/main/java/org/apache/drill/exec/store/plugin/PcapFormatConfig.java
##########
@@ -29,18 +29,23 @@
 import com.fasterxml.jackson.annotation.JsonProperty;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 
-@JsonTypeName(PcapngFormatConfig.NAME)
+@JsonTypeName(PcapFormatConfig.NAME)
 @JsonInclude(JsonInclude.Include.NON_DEFAULT)
-public class PcapngFormatConfig implements FormatPluginConfig {
+public class PcapFormatConfig implements FormatPluginConfig {
+  private static final List<String> DEFAULT_EXTNS = ImmutableList.of("pcap", "pcapng");
 
-  public static final String NAME = "pcapng";
+  public static final String NAME = "pcap";
   private final List<String> extensions;
   private final boolean stat;
+  private final boolean sessionizeTCPStreams;
 
   @JsonCreator
-  public PcapngFormatConfig(@JsonProperty("extensions") List<String> extensions, @JsonProperty("stat") boolean stat) {
-    this.extensions = extensions == null ? ImmutableList.of(PcapngFormatConfig.NAME) : ImmutableList.copyOf(extensions);
+  public PcapFormatConfig(@JsonProperty("extensions") List<String> extensions,
+                          @JsonProperty("stat") boolean stat,
+                          @JsonProperty("sessionizeTCPStreams") Boolean sessionizeTCPStreams) {

Review comment:
       Thanks for the explanation. If the property already existed, then we must leave it to preserve backward compatibility. I guess we should have caught the issue when the property was originally added.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org