You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Matthew Ouyang <ma...@gmail.com> on 2020/08/02 13:17:13 UTC
Development on Windows
Hello,
I am working on a small patch on a Windows environment. But I'm unable to
build the entire project end-to-end. Here are a few of the errors I am
receiving.
Is there any sort of setup, Windows-specific or otherwise, required in
order to build the project completely?
Matt
./gradlew build
FAILURE: Build completed with 5 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':beam-test-infra-metrics:spotlessGroovyGradle'.
> The following files had format violations:
.test-infra\metrics\build.gradle
@@ -1,120 +1,120 @@
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.nio.file.FileSystemException:
C:\Users\maouyan\AppData\Local\Temp\junit1348014844526437497\BigQueryExtractTemp\99a851333c89412b8cf0fe0d53a7c084\000000000000.avro:
The process cannot access the file because it is being used by another
process.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Class org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOReadTest
all > org.apache.beam.sdk.io.gcp.bigquery > BigQueryIOReadTest
testReadFromTableWithTemplateCompatibility
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.nio.file.FileSystemException:
C:\Users\maouyan\AppData\Local\Temp\junit1348014844526437497\BigQueryExtractTemp\99a851333c89412b8cf0fe0d53a7c084\000000000000.avro:
The process cannot access the file because it is being used by another
process.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Class org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest
all > org.apache.beam.sdk.io.gcp.bigquery > BigQueryIOWriteTest
testWritePartitionSinglePartition
java.nio.file.InvalidPathException: Illegal char <:> at index 10:
project-id:dataset-id.tables00000_files00000
Re: Development on Windows
Posted by Matthew Ouyang <ma...@gmail.com>.
Hi Luke,
Sorry for the late reply. Things got busy at work. I took an initial try
at this but the issue still persisted. When I get some time, I'll try
again.
Matt
On Mon, Aug 3, 2020 at 1:38 PM Luke Cwik <lc...@google.com> wrote:
> It looks like Windows file locking is getting in your way since one gradle
> task is reading a file which is produced as a temporary file that is part
> of another gradle test task and that test task is trying to delete it.
>
> Try running the build with --no-parallel to disable parallel builds[1].
> This will make your build a lot slower though but will avoid the issue of
> having multiple gradle tasks stepping on each other's toes due to file
> locking.
>
> For a longer term fix, it seems as though spotless shouldn't care about
> these temp files being produced so feel free to contribute a patch which
> removes spotless from checking anything which isn't a source file.
>
> 1:
> https://docs.gradle.org/current/userguide/multi_project_builds.html#sec:parallel_execution
>
>
>
> On Sun, Aug 2, 2020 at 6:17 AM Matthew Ouyang <ma...@gmail.com>
> wrote:
>
>> Hello,
>>
>> I am working on a small patch on a Windows environment. But I'm unable
>> to build the entire project end-to-end. Here are a few of the errors I am
>> receiving.
>>
>> Is there any sort of setup, Windows-specific or otherwise, required in
>> order to build the project completely?
>>
>> Matt
>>
>>
>> ./gradlew build
>> FAILURE: Build completed with 5 failures.
>>
>> 1: Task failed with an exception.
>> -----------
>> * What went wrong:
>> Execution failed for task ':beam-test-infra-metrics:spotlessGroovyGradle'.
>> > The following files had format violations:
>> .test-infra\metrics\build.gradle
>> @@ -1,120 +1,120 @@
>>
>>
>>
>> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
>> java.nio.file.FileSystemException:
>> C:\Users\maouyan\AppData\Local\Temp\junit1348014844526437497\BigQueryExtractTemp\99a851333c89412b8cf0fe0d53a7c084\000000000000.avro:
>> The process cannot access the file because it is being used by another
>> process.
>>
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> Class org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOReadTest
>> all > org.apache.beam.sdk.io.gcp.bigquery > BigQueryIOReadTest
>>
>> testReadFromTableWithTemplateCompatibility
>> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
>> java.nio.file.FileSystemException:
>> C:\Users\maouyan\AppData\Local\Temp\junit1348014844526437497\BigQueryExtractTemp\99a851333c89412b8cf0fe0d53a7c084\000000000000.avro:
>> The process cannot access the file because it is being used by another
>> process.
>>
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> Class org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest
>> all > org.apache.beam.sdk.io.gcp.bigquery > BigQueryIOWriteTest
>>
>> testWritePartitionSinglePartition
>> java.nio.file.InvalidPathException: Illegal char <:> at index 10:
>> project-id:dataset-id.tables00000_files00000
>>
>
Re: Development on Windows
Posted by Luke Cwik <lc...@google.com>.
It looks like Windows file locking is getting in your way since one gradle
task is reading a file which is produced as a temporary file that is part
of another gradle test task and that test task is trying to delete it.
Try running the build with --no-parallel to disable parallel builds[1].
This will make your build a lot slower though but will avoid the issue of
having multiple gradle tasks stepping on each other's toes due to file
locking.
For a longer term fix, it seems as though spotless shouldn't care about
these temp files being produced so feel free to contribute a patch which
removes spotless from checking anything which isn't a source file.
1:
https://docs.gradle.org/current/userguide/multi_project_builds.html#sec:parallel_execution
On Sun, Aug 2, 2020 at 6:17 AM Matthew Ouyang <ma...@gmail.com>
wrote:
> Hello,
>
> I am working on a small patch on a Windows environment. But I'm unable to
> build the entire project end-to-end. Here are a few of the errors I am
> receiving.
>
> Is there any sort of setup, Windows-specific or otherwise, required in
> order to build the project completely?
>
> Matt
>
>
> ./gradlew build
> FAILURE: Build completed with 5 failures.
>
> 1: Task failed with an exception.
> -----------
> * What went wrong:
> Execution failed for task ':beam-test-infra-metrics:spotlessGroovyGradle'.
> > The following files had format violations:
> .test-infra\metrics\build.gradle
> @@ -1,120 +1,120 @@
>
>
>
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.nio.file.FileSystemException:
> C:\Users\maouyan\AppData\Local\Temp\junit1348014844526437497\BigQueryExtractTemp\99a851333c89412b8cf0fe0d53a7c084\000000000000.avro:
> The process cannot access the file because it is being used by another
> process.
>
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Class org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOReadTest
> all > org.apache.beam.sdk.io.gcp.bigquery > BigQueryIOReadTest
>
> testReadFromTableWithTemplateCompatibility
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.nio.file.FileSystemException:
> C:\Users\maouyan\AppData\Local\Temp\junit1348014844526437497\BigQueryExtractTemp\99a851333c89412b8cf0fe0d53a7c084\000000000000.avro:
> The process cannot access the file because it is being used by another
> process.
>
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Class org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest
> all > org.apache.beam.sdk.io.gcp.bigquery > BigQueryIOWriteTest
>
> testWritePartitionSinglePartition
> java.nio.file.InvalidPathException: Illegal char <:> at index 10:
> project-id:dataset-id.tables00000_files00000
>