You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Andrew Sherman via Review Board <no...@reviews.apache.org> on 2018/05/11 23:38:43 UTC
Re: Review Request 67023: HIVE-18117: Add a new Test Driver
"TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs
directories that employ Erasure Coding.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/
-----------------------------------------------------------
(Updated May 11, 2018, 11:38 p.m.)
Review request for hive and Sahil Takiar.
Repository: hive-git
Description
-------
TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor "ErasureProcessor"
which allows .q files to contain Erasure Coding commands similar to those provided
by the hdfs ec command
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
The Erasure Coding functionality is exposed through a new shim "HdfsFileErasureCodingPolicy".
At this stage there are two .q files:
erasure_commnds.q (a simple test to show ERASURE commands can run on local fs via
TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
erasure_simple.q (which does some trivial queries to demonstrate basic functionality).
More tests will come in future commits.
Diffs (updated)
-----
itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java PRE-CREATION
itests/src/test/resources/testconfiguration.properties cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 6628336807b06cab49063673be0d8e9c5b5a7101
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 750fc69c5f210ca8f7bfe81b82ee9e3001fc07ba
ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 3d47991b603c94c8da2106e67192c8513ef783a7
ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 56c7516ecfaf2421b0f3d3a188d05f38715b25b2
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 9f65a771f95a7c0bd3fdb4e56e47c0fc70235850
ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java de43c2866f64e2ed5c74eab450de28f1a79248dc
ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION
ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION
ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION
ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out PRE-CREATION
ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out PRE-CREATION
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java ec06a88dc21346473bec6589c703167d50e3b367
shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java b89081761780bf1f305d0196bb94bb0b54f7184f
testutils/ptest2/conf/deployed/master-mr2.properties 7edc307f85744d60d322ad8087164625677fc230
Diff: https://reviews.apache.org/r/67023/diff/2/
Changes: https://reviews.apache.org/r/67023/diff/1-2/
Testing
-------
Thanks,
Andrew Sherman
Re: Review Request 67023: HIVE-18117: Add a new Test Driver
"TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs
directories that employ Erasure Coding.
Posted by Andrew Sherman via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/
-----------------------------------------------------------
(Updated May 14, 2018, 9:44 p.m.)
Review request for hive and Sahil Takiar.
Repository: hive-git
Description
-------
TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor "ErasureProcessor"
which allows .q files to contain Erasure Coding commands similar to those provided
by the hdfs ec command
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
The Erasure Coding functionality is exposed through a new shim "HdfsFileErasureCodingPolicy".
At this stage there are two .q files:
erasure_commnds.q (a simple test to show ERASURE commands can run on local fs via
TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
erasure_simple.q (which does some trivial queries to demonstrate basic functionality).
More tests will come in future commits.
Diffs (updated)
-----
itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java PRE-CREATION
itests/src/test/resources/testconfiguration.properties cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 16571b3ff3288dfb99fbca570452592cc1650f9a
ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 3d47991b603c94c8da2106e67192c8513ef783a7
ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 56c7516ecfaf2421b0f3d3a188d05f38715b25b2
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 89129f99fe63f0384aff965ad665770d11e9af04
ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java de43c2866f64e2ed5c74eab450de28f1a79248dc
ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION
ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION
ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION
ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out PRE-CREATION
ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out PRE-CREATION
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java ec06a88dc21346473bec6589c703167d50e3b367
shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java b89081761780bf1f305d0196bb94bb0b54f7184f
testutils/ptest2/conf/deployed/master-mr2.properties 7edc307f85744d60d322ad8087164625677fc230
Diff: https://reviews.apache.org/r/67023/diff/3/
Changes: https://reviews.apache.org/r/67023/diff/2-3/
Testing
-------
Thanks,
Andrew Sherman
Re: Review Request 67023: HIVE-18117: Add a new Test Driver
"TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs
directories that employ Erasure Coding.
Posted by Andrew Sherman via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202981
-----------------------------------------------------------
ql/src/test/queries/clientpositive/erasure_simple.q
Lines 33 (patched)
<https://reviews.apache.org/r/67023/#comment285041>
should be 'select key, value from src;'
- Andrew Sherman
On May 11, 2018, 11:38 p.m., Andrew Sherman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> -----------------------------------------------------------
>
> (Updated May 11, 2018, 11:38 p.m.)
>
>
> Review request for hive and Sahil Takiar.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic functionality).
> More tests will come in future commits.
>
>
> Diffs
> -----
>
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java PRE-CREATION
> itests/src/test/resources/testconfiguration.properties cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 6628336807b06cab49063673be0d8e9c5b5a7101
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 750fc69c5f210ca8f7bfe81b82ee9e3001fc07ba
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 3d47991b603c94c8da2106e67192c8513ef783a7
> ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 56c7516ecfaf2421b0f3d3a188d05f38715b25b2
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 9f65a771f95a7c0bd3fdb4e56e47c0fc70235850
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java de43c2866f64e2ed5c74eab450de28f1a79248dc
> ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION
> ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION
> ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION
> ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out PRE-CREATION
> ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out PRE-CREATION
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java ec06a88dc21346473bec6589c703167d50e3b367
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java b89081761780bf1f305d0196bb94bb0b54f7184f
> testutils/ptest2/conf/deployed/master-mr2.properties 7edc307f85744d60d322ad8087164625677fc230
>
>
> Diff: https://reviews.apache.org/r/67023/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Andrew Sherman
>
>
Re: Review Request 67023: HIVE-18117: Add a new Test Driver
"TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs
directories that employ Erasure Coding.
Posted by Andrew Sherman via Review Board <no...@reviews.apache.org>.
> On May 14, 2018, 4:57 p.m., Sahil Takiar wrote:
> > itests/src/test/resources/testconfiguration.properties
> > Line 1693 (original), 1693 (patched)
> > <https://reviews.apache.org/r/67023/diff/1-2/?file=2018286#file2018286line1693>
> >
> > why would we want the ec commands to work outside the `TestErasureCodingHDFSCliDriver`?
>
> Andrew Sherman wrote:
> So you could run an ec test in TestCliDriver (sorry reviewboard lost my ealrier reply)
>
> Sahil Takiar wrote:
> why would you want to do that? whats the use case? shouldn't `TestErasureCodingHDFSCliDriver` encapsulate all EC-related tests? plus I'm not sure how you could run any EC commands against a local filesystem, wouldn't they all be no-ops?
They would. When I'm developing an EC test I may sometimes want to run the test in TestCLiDriver with erasure commands being no-ops as a way to validate the script. Myabe this is weird, but I've been doing it.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review203040
-----------------------------------------------------------
On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> -----------------------------------------------------------
>
> (Updated May 14, 2018, 9:44 p.m.)
>
>
> Review request for hive and Sahil Takiar.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic functionality).
> More tests will come in future commits.
>
>
> Diffs
> -----
>
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java PRE-CREATION
> itests/src/test/resources/testconfiguration.properties cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 16571b3ff3288dfb99fbca570452592cc1650f9a
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 3d47991b603c94c8da2106e67192c8513ef783a7
> ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 56c7516ecfaf2421b0f3d3a188d05f38715b25b2
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 89129f99fe63f0384aff965ad665770d11e9af04
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java de43c2866f64e2ed5c74eab450de28f1a79248dc
> ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION
> ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION
> ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION
> ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out PRE-CREATION
> ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out PRE-CREATION
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java ec06a88dc21346473bec6589c703167d50e3b367
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java b89081761780bf1f305d0196bb94bb0b54f7184f
> testutils/ptest2/conf/deployed/master-mr2.properties 7edc307f85744d60d322ad8087164625677fc230
>
>
> Diff: https://reviews.apache.org/r/67023/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Andrew Sherman
>
>
Re: Review Request 67023: HIVE-18117: Add a new Test Driver
"TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs
directories that employ Erasure Coding.
Posted by Sahil Takiar <ta...@gmail.com>.
> On May 14, 2018, 4:57 p.m., Sahil Takiar wrote:
> > itests/src/test/resources/testconfiguration.properties
> > Line 1693 (original), 1693 (patched)
> > <https://reviews.apache.org/r/67023/diff/1-2/?file=2018286#file2018286line1693>
> >
> > why would we want the ec commands to work outside the `TestErasureCodingHDFSCliDriver`?
>
> Andrew Sherman wrote:
> So you could run an ec test in TestCliDriver (sorry reviewboard lost my ealrier reply)
why would you want to do that? whats the use case? shouldn't `TestErasureCodingHDFSCliDriver` encapsulate all EC-related tests? plus I'm not sure how you could run any EC commands against a local filesystem, wouldn't they all be no-ops?
- Sahil
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review203040
-----------------------------------------------------------
On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> -----------------------------------------------------------
>
> (Updated May 14, 2018, 9:44 p.m.)
>
>
> Review request for hive and Sahil Takiar.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic functionality).
> More tests will come in future commits.
>
>
> Diffs
> -----
>
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java PRE-CREATION
> itests/src/test/resources/testconfiguration.properties cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 16571b3ff3288dfb99fbca570452592cc1650f9a
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 3d47991b603c94c8da2106e67192c8513ef783a7
> ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 56c7516ecfaf2421b0f3d3a188d05f38715b25b2
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 89129f99fe63f0384aff965ad665770d11e9af04
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java de43c2866f64e2ed5c74eab450de28f1a79248dc
> ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION
> ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION
> ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION
> ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out PRE-CREATION
> ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out PRE-CREATION
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java ec06a88dc21346473bec6589c703167d50e3b367
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java b89081761780bf1f305d0196bb94bb0b54f7184f
> testutils/ptest2/conf/deployed/master-mr2.properties 7edc307f85744d60d322ad8087164625677fc230
>
>
> Diff: https://reviews.apache.org/r/67023/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Andrew Sherman
>
>
Re: Review Request 67023: HIVE-18117: Add a new Test Driver
"TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs
directories that employ Erasure Coding.
Posted by Andrew Sherman via Review Board <no...@reviews.apache.org>.
> On May 14, 2018, 4:57 p.m., Sahil Takiar wrote:
> > itests/src/test/resources/testconfiguration.properties
> > Line 1693 (original), 1693 (patched)
> > <https://reviews.apache.org/r/67023/diff/1-2/?file=2018286#file2018286line1693>
> >
> > why would we want the ec commands to work outside the `TestErasureCodingHDFSCliDriver`?
So you could run an ec test in TestCliDriver (sorry reviewboard lost my ealrier reply)
> On May 14, 2018, 4:57 p.m., Sahil Takiar wrote:
> > ql/src/test/queries/clientpositive/erasure_commands.q
> > Lines 2 (patched)
> > <https://reviews.apache.org/r/67023/diff/2/?file=2021052#file2021052line2>
> >
> > why would we want to run this on the local fs?
I'm imaginging wanting to run an ec test in TestCliDriver without changing it (sorry reviewboard lost my ealrier reply)
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review203040
-----------------------------------------------------------
On May 11, 2018, 11:38 p.m., Andrew Sherman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> -----------------------------------------------------------
>
> (Updated May 11, 2018, 11:38 p.m.)
>
>
> Review request for hive and Sahil Takiar.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic functionality).
> More tests will come in future commits.
>
>
> Diffs
> -----
>
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java PRE-CREATION
> itests/src/test/resources/testconfiguration.properties cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 6628336807b06cab49063673be0d8e9c5b5a7101
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 750fc69c5f210ca8f7bfe81b82ee9e3001fc07ba
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 3d47991b603c94c8da2106e67192c8513ef783a7
> ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 56c7516ecfaf2421b0f3d3a188d05f38715b25b2
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 9f65a771f95a7c0bd3fdb4e56e47c0fc70235850
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java de43c2866f64e2ed5c74eab450de28f1a79248dc
> ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION
> ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION
> ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION
> ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out PRE-CREATION
> ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out PRE-CREATION
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java ec06a88dc21346473bec6589c703167d50e3b367
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java b89081761780bf1f305d0196bb94bb0b54f7184f
> testutils/ptest2/conf/deployed/master-mr2.properties 7edc307f85744d60d322ad8087164625677fc230
>
>
> Diff: https://reviews.apache.org/r/67023/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Andrew Sherman
>
>
Re: Review Request 67023: HIVE-18117: Add a new Test Driver
"TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs
directories that employ Erasure Coding.
Posted by Sahil Takiar <ta...@gmail.com>.
> On May 14, 2018, 4:57 p.m., Sahil Takiar wrote:
> > itests/src/test/resources/testconfiguration.properties
> > Line 1693 (original), 1693 (patched)
> > <https://reviews.apache.org/r/67023/diff/1-2/?file=2018286#file2018286line1693>
> >
> > why would we want the ec commands to work outside the `TestErasureCodingHDFSCliDriver`?
>
> Andrew Sherman wrote:
> So you could run an ec test in TestCliDriver (sorry reviewboard lost my ealrier reply)
>
> Sahil Takiar wrote:
> why would you want to do that? whats the use case? shouldn't `TestErasureCodingHDFSCliDriver` encapsulate all EC-related tests? plus I'm not sure how you could run any EC commands against a local filesystem, wouldn't they all be no-ops?
>
> Andrew Sherman wrote:
> They would. When I'm developing an EC test I may sometimes want to run the test in TestCLiDriver with erasure commands being no-ops as a way to validate the script. Myabe this is weird, but I've been doing it.
I see, makes sense.
- Sahil
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review203040
-----------------------------------------------------------
On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> -----------------------------------------------------------
>
> (Updated May 14, 2018, 9:44 p.m.)
>
>
> Review request for hive and Sahil Takiar.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic functionality).
> More tests will come in future commits.
>
>
> Diffs
> -----
>
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java PRE-CREATION
> itests/src/test/resources/testconfiguration.properties cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 16571b3ff3288dfb99fbca570452592cc1650f9a
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 3d47991b603c94c8da2106e67192c8513ef783a7
> ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 56c7516ecfaf2421b0f3d3a188d05f38715b25b2
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 89129f99fe63f0384aff965ad665770d11e9af04
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java de43c2866f64e2ed5c74eab450de28f1a79248dc
> ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION
> ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION
> ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION
> ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out PRE-CREATION
> ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out PRE-CREATION
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java ec06a88dc21346473bec6589c703167d50e3b367
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java b89081761780bf1f305d0196bb94bb0b54f7184f
> testutils/ptest2/conf/deployed/master-mr2.properties 7edc307f85744d60d322ad8087164625677fc230
>
>
> Diff: https://reviews.apache.org/r/67023/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Andrew Sherman
>
>
Re: Review Request 67023: HIVE-18117: Add a new Test Driver
"TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs
directories that employ Erasure Coding.
Posted by Sahil Takiar <ta...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review203040
-----------------------------------------------------------
itests/src/test/resources/testconfiguration.properties
Line 1693 (original), 1693 (patched)
<https://reviews.apache.org/r/67023/#comment285103>
why would we want the ec commands to work outside the `TestErasureCodingHDFSCliDriver`?
ql/src/test/queries/clientpositive/erasure_commands.q
Lines 2 (patched)
<https://reviews.apache.org/r/67023/#comment285102>
why would we want to run this on the local fs?
- Sahil Takiar
On May 11, 2018, 11:38 p.m., Andrew Sherman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> -----------------------------------------------------------
>
> (Updated May 11, 2018, 11:38 p.m.)
>
>
> Review request for hive and Sahil Takiar.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic functionality).
> More tests will come in future commits.
>
>
> Diffs
> -----
>
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java PRE-CREATION
> itests/src/test/resources/testconfiguration.properties cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 6628336807b06cab49063673be0d8e9c5b5a7101
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 750fc69c5f210ca8f7bfe81b82ee9e3001fc07ba
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 3d47991b603c94c8da2106e67192c8513ef783a7
> ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 56c7516ecfaf2421b0f3d3a188d05f38715b25b2
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 9f65a771f95a7c0bd3fdb4e56e47c0fc70235850
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java de43c2866f64e2ed5c74eab450de28f1a79248dc
> ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION
> ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION
> ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION
> ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out PRE-CREATION
> ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out PRE-CREATION
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java ec06a88dc21346473bec6589c703167d50e3b367
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java b89081761780bf1f305d0196bb94bb0b54f7184f
> testutils/ptest2/conf/deployed/master-mr2.properties 7edc307f85744d60d322ad8087164625677fc230
>
>
> Diff: https://reviews.apache.org/r/67023/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Andrew Sherman
>
>