You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by lresende <gi...@git.apache.org> on 2016/01/22 21:35:25 UTC
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user lresende commented on the pull request:
https://github.com/apache/spark/pull/9893#issuecomment-174039005
@JoshRosen For this PR to work, there is a need to manually deploy the JDBC driver as described on the pom. This is similar to the requirement that in order to run these tests, people must have docker installed.
As for deploying the driver into maven or something, I am trying to get that approved, but for now people can download the driver manually via the link described in the pom.
As for running these, I have ran it on our Jenkins environment, and see the results of the DB2 Integration tests below.
Could you please review this, help with the Jenkins slaves configuration, and merge so we can move forward with the rest of the DB2 dialect prs waiting on this.
Db2IntegrationSuite:
16/01/22 12:26:56 INFO Slf4jLogger: Slf4jLogger started
16/01/22 12:26:56 INFO Remoting: Starting remoting
16/01/22 12:26:56 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:33460]
16/01/22 12:26:56 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 33460.
16/01/22 12:26:56 INFO SparkEnv: Registering MapOutputTracker
16/01/22 12:26:56 INFO SparkEnv: Registering BlockManagerMaster
16/01/22 12:26:56 INFO DiskBlockManager: Created local directory at /a/workspace/Spark-Build_and_Test_with_docker/docker-integration-tests/target/tmp/blockmgr-73d7995e-9327-46d8-9a4e-3a9b991e2782
16/01/22 12:26:56 INFO MemoryStore: MemoryStore started with capacity 2.0 GB
16/01/22 12:26:56 INFO SparkEnv: Registering OutputCommitCoordinator
16/01/22 12:26:56 INFO Executor: Starting executor ID driver on host localhost
16/01/22 12:26:56 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 36866.
16/01/22 12:26:56 INFO NettyBlockTransferService: Server created on 36866
16/01/22 12:26:56 INFO BlockManagerMaster: Trying to register BlockManager
16/01/22 12:26:56 INFO BlockManagerMasterEndpoint: Registering block manager localhost:36866 with 2.0 GB RAM, BlockManagerId(driver, localhost, 36866)
16/01/22 12:26:56 INFO BlockManagerMaster: Registered BlockManager
16/01/22 12:26:56 INFO DefaultDockerClient: Creating container with ContainerConfig: ContainerConfig{hostname=null, domainname=null, username=null, attachStdin=null, attachStdout=null, attachStderr=null, portSpecs=null, exposedPorts=[50000/tcp], tty=null, openStdin=null, stdinOnce=null, env=[DB2INST1_PASSWORD=rootpass, LICENSE=accept], cmd=[db2start], image=lresende/db2express-c:10.5.0.5-3.10.0, volumes=null, workingDir=null, entrypoint=null, networkDisabled=false, onBuild=null, labels=null, macAddress=null, hostConfig=HostConfig{binds=null, containerIDFile=null, lxcConf=null, privileged=null, portBindings={50000/tcp=[PortBinding{hostIp=9.30.122.152, hostPort=51408}]}, links=null, publishAllPorts=null, dns=null, dnsSearch=null, extraHosts=null, volumesFrom=null, networkMode=bridge, securityOpt=null, memory=null, memorySwap=null, cpuShares=null, cpusetCpus=null, cpuQuota=null, cgroupParent=null}}
16/01/22 12:26:58 INFO DefaultDockerClient: Starting container with Id: bc6620ff6e3c41521e48752164ded0290e4d97c30ce1c30d68436fa45ea6863c
16/01/22 12:27:05 INFO Db2IntegrationSuite:
===== TEST OUTPUT FOR o.a.s.sql.jdbc.Db2IntegrationSuite: 'Basic test' =====
16/01/22 12:27:05 INFO SparkContext: Starting job: apply at Transformer.scala:22
16/01/22 12:27:05 INFO DAGScheduler: Got job 0 (apply at Transformer.scala:22) with 1 output partitions
16/01/22 12:27:05 INFO DAGScheduler: Final stage: ResultStage 0 (apply at Transformer.scala:22)
16/01/22 12:27:05 INFO DAGScheduler: Parents of final stage: List()
16/01/22 12:27:05 INFO DAGScheduler: Missing parents: List()
16/01/22 12:27:05 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at apply at Transformer.scala:22), which has no missing parents
16/01/22 12:27:05 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 6.6 KB, free 6.6 KB)
16/01/22 12:27:05 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.3 KB, free 9.9 KB)
16/01/22 12:27:05 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:36866 (size: 3.3 KB, free: 2.0 GB)
16/01/22 12:27:05 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1010
16/01/22 12:27:05 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at apply at Transformer.scala:22)
16/01/22 12:27:05 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
16/01/22 12:27:05 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 1972 bytes)
16/01/22 12:27:05 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/01/22 12:27:05 INFO JDBCRDD: closed connection
16/01/22 12:27:05 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1476 bytes result sent to driver
16/01/22 12:27:05 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 289 ms on localhost (1/1)
16/01/22 12:27:05 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
16/01/22 12:27:05 INFO DAGScheduler: ResultStage 0 (apply at Transformer.scala:22) finished in 0.290 s
16/01/22 12:27:05 INFO DAGScheduler: Job 0 finished: apply at Transformer.scala:22, took 0.299543 s
16/01/22 12:27:05 INFO Db2IntegrationSuite:
===== FINISHED o.a.s.sql.jdbc.Db2IntegrationSuite: 'Basic test' =====
16/01/22 12:27:05 INFO Db2IntegrationSuite:
===== TEST OUTPUT FOR o.a.s.sql.jdbc.Db2IntegrationSuite: 'Numeric types' =====
- Basic test
16/01/22 12:27:06 INFO SparkContext: Starting job: apply at Transformer.scala:22
16/01/22 12:27:06 INFO DAGScheduler: Got job 1 (apply at Transformer.scala:22) with 1 output partitions
16/01/22 12:27:06 INFO DAGScheduler: Final stage: ResultStage 1 (apply at Transformer.scala:22)
16/01/22 12:27:06 INFO DAGScheduler: Parents of final stage: List()
16/01/22 12:27:06 INFO DAGScheduler: Missing parents: List()
16/01/22 12:27:06 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[5] at apply at Transformer.scala:22), which has no missing parents
16/01/22 12:27:06 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 7.4 KB, free 17.3 KB)
16/01/22 12:27:06 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.7 KB, free 21.0 KB)
16/01/22 12:27:06 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:36866 (size: 3.7 KB, free: 2.0 GB)
16/01/22 12:27:06 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1010
16/01/22 12:27:06 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[5] at apply at Transformer.scala:22)
16/01/22 12:27:06 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
16/01/22 12:27:06 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, partition 0,PROCESS_LOCAL, 1972 bytes)
16/01/22 12:27:06 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
16/01/22 12:27:06 INFO CodeGenerator: Code generated in 12.625054 ms
16/01/22 12:27:06 INFO JDBCRDD: closed connection
16/01/22 12:27:06 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1467 bytes result sent to driver
16/01/22 12:27:06 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 392 ms on localhost (1/1)
16/01/22 12:27:06 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
16/01/22 12:27:06 INFO DAGScheduler: ResultStage 1 (apply at Transformer.scala:22) finished in 0.393 s
16/01/22 12:27:06 INFO DAGScheduler: Job 1 finished: apply at Transformer.scala:22, took 0.400140 s
16/01/22 12:27:06 INFO Db2IntegrationSuite:
===== FINISHED o.a.s.sql.jdbc.Db2IntegrationSuite: 'Numeric types' =====
- Numeric types
16/01/22 12:27:06 INFO Db2IntegrationSuite:
===== TEST OUTPUT FOR o.a.s.sql.jdbc.Db2IntegrationSuite: 'Date types' =====
16/01/22 12:27:06 INFO SparkContext: Starting job: apply at Transformer.scala:22
16/01/22 12:27:06 INFO DAGScheduler: Got job 2 (apply at Transformer.scala:22) with 1 output partitions
16/01/22 12:27:06 INFO DAGScheduler: Final stage: ResultStage 2 (apply at Transformer.scala:22)
16/01/22 12:27:06 INFO DAGScheduler: Parents of final stage: List()
16/01/22 12:27:06 INFO DAGScheduler: Missing parents: List()
16/01/22 12:27:06 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[8] at apply at Transformer.scala:22), which has no missing parents
16/01/22 12:27:06 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 6.7 KB, free 27.6 KB)
16/01/22 12:27:06 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 3.4 KB, free 31.0 KB)
16/01/22 12:27:06 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:36866 (size: 3.4 KB, free: 2.0 GB)
16/01/22 12:27:06 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1010
16/01/22 12:27:06 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[8] at apply at Transformer.scala:22)
16/01/22 12:27:06 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
16/01/22 12:27:06 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, partition 0,PROCESS_LOCAL, 1972 bytes)
16/01/22 12:27:06 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
16/01/22 12:27:07 INFO CodeGenerator: Code generated in 11.399165 ms
16/01/22 12:27:07 INFO JDBCRDD: closed connection
16/01/22 12:27:07 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 1427 bytes result sent to driver
16/01/22 12:27:07 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 315 ms on localhost (1/1)
16/01/22 12:27:07 INFO DAGScheduler: ResultStage 2 (apply at Transformer.scala:22) finished in 0.315 s
16/01/22 12:27:07 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
16/01/22 12:27:07 INFO DAGScheduler: Job 2 finished: apply at Transformer.scala:22, took 0.321779 s
16/01/22 12:27:07 INFO Db2IntegrationSuite:
===== FINISHED o.a.s.sql.jdbc.Db2IntegrationSuite: 'Date types' =====
- Date types
16/01/22 12:27:07 INFO Db2IntegrationSuite:
===== TEST OUTPUT FOR o.a.s.sql.jdbc.Db2IntegrationSuite: 'String types' =====
16/01/22 12:27:07 INFO SparkContext: Starting job: apply at Transformer.scala:22
16/01/22 12:27:07 INFO DAGScheduler: Got job 3 (apply at Transformer.scala:22) with 1 output partitions
16/01/22 12:27:07 INFO DAGScheduler: Final stage: ResultStage 3 (apply at Transformer.scala:22)
16/01/22 12:27:07 INFO DAGScheduler: Parents of final stage: List()
16/01/22 12:27:07 INFO DAGScheduler: Missing parents: List()
16/01/22 12:27:07 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[11] at apply at Transformer.scala:22), which has no missing parents
16/01/22 12:27:07 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 6.7 KB, free 37.7 KB)
16/01/22 12:27:07 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 3.4 KB, free 41.1 KB)
16/01/22 12:27:07 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:36866 (size: 3.4 KB, free: 2.0 GB)
16/01/22 12:27:07 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1010
16/01/22 12:27:07 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[11] at apply at Transformer.scala:22)
16/01/22 12:27:07 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
16/01/22 12:27:07 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, localhost, partition 0,PROCESS_LOCAL, 1972 bytes)
16/01/22 12:27:07 INFO Executor: Running task 0.0 in stage 3.0 (TID 3)
16/01/22 12:27:07 INFO CodeGenerator: Code generated in 8.298278 ms
16/01/22 12:27:07 INFO JDBCRDD: closed connection
16/01/22 12:27:07 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 1475 bytes result sent to driver
16/01/22 12:27:07 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 360 ms on localhost (1/1)
16/01/22 12:27:07 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
16/01/22 12:27:07 INFO DAGScheduler: ResultStage 3 (apply at Transformer.scala:22) finished in 0.361 s
16/01/22 12:27:07 INFO DAGScheduler: Job 3 finished: apply at Transformer.scala:22, took 0.368710 s
16/01/22 12:27:07 INFO Db2IntegrationSuite:
===== FINISHED o.a.s.sql.jdbc.Db2IntegrationSuite: 'String types' =====
- String types
16/01/22 12:27:07 INFO Db2IntegrationSuite:
===== TEST OUTPUT FOR o.a.s.sql.jdbc.Db2IntegrationSuite: 'Basic write test' =====
16/01/22 12:27:09 INFO SparkContext: Starting job: apply at Transformer.scala:22
16/01/22 12:27:09 INFO DAGScheduler: Got job 4 (apply at Transformer.scala:22) with 1 output partitions
16/01/22 12:27:09 INFO DAGScheduler: Final stage: ResultStage 4 (apply at Transformer.scala:22)
16/01/22 12:27:09 INFO DAGScheduler: Parents of final stage: List()
16/01/22 12:27:09 INFO DAGScheduler: Missing parents: List()
16/01/22 12:27:09 INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[14] at apply at Transformer.scala:22), which has no missing parents
16/01/22 12:27:09 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 7.1 KB, free 48.2 KB)
16/01/22 12:27:09 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 3.6 KB, free 51.9 KB)
16/01/22 12:27:09 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:36866 (size: 3.6 KB, free: 2.0 GB)
16/01/22 12:27:09 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1010
16/01/22 12:27:09 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[14] at apply at Transformer.scala:22)
16/01/22 12:27:09 INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
16/01/22 12:27:09 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, localhost, partition 0,PROCESS_LOCAL, 1972 bytes)
16/01/22 12:27:09 INFO Executor: Running task 0.0 in stage 4.0 (TID 4)
16/01/22 12:27:09 INFO JDBCRDD: closed connection
16/01/22 12:27:09 INFO Executor: Finished task 0.0 in stage 4.0 (TID 4). 1165 bytes result sent to driver
16/01/22 12:27:09 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 387 ms on localhost (1/1)
16/01/22 12:27:09 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
16/01/22 12:27:09 INFO DAGScheduler: ResultStage 4 (apply at Transformer.scala:22) finished in 0.388 s
16/01/22 12:27:09 INFO DAGScheduler: Job 4 finished: apply at Transformer.scala:22, took 0.394228 s
16/01/22 12:27:10 INFO SparkContext: Starting job: apply at Transformer.scala:22
16/01/22 12:27:10 INFO DAGScheduler: Got job 5 (apply at Transformer.scala:22) with 1 output partitions
16/01/22 12:27:10 INFO DAGScheduler: Final stage: ResultStage 5 (apply at Transformer.scala:22)
16/01/22 12:27:10 INFO DAGScheduler: Parents of final stage: List()
16/01/22 12:27:10 INFO DAGScheduler: Missing parents: List()
16/01/22 12:27:10 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[17] at apply at Transformer.scala:22), which has no missing parents
16/01/22 12:27:10 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 7.2 KB, free 59.1 KB)
16/01/22 12:27:10 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 3.7 KB, free 62.8 KB)
16/01/22 12:27:10 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:36866 (size: 3.7 KB, free: 2.0 GB)
16/01/22 12:27:10 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1010
16/01/22 12:27:10 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[17] at apply at Transformer.scala:22)
16/01/22 12:27:10 INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
16/01/22 12:27:10 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 5, localhost, partition 0,PROCESS_LOCAL, 1972 bytes)
16/01/22 12:27:10 INFO Executor: Running task 0.0 in stage 5.0 (TID 5)
16/01/22 12:27:10 INFO JDBCRDD: closed connection
16/01/22 12:27:10 INFO Executor: Finished task 0.0 in stage 5.0 (TID 5). 1165 bytes result sent to driver
16/01/22 12:27:10 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 5) in 433 ms on localhost (1/1)
16/01/22 12:27:10 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool
16/01/22 12:27:10 INFO DAGScheduler: ResultStage 5 (apply at Transformer.scala:22) finished in 0.435 s
16/01/22 12:27:10 INFO DAGScheduler: Job 5 finished: apply at Transformer.scala:22, took 0.441109 s
16/01/22 12:27:10 INFO Db2IntegrationSuite:
===== FINISHED o.a.s.sql.jdbc.Db2IntegrationSuite: 'Basic write test' =====
- Basic write test
16/01/22 12:27:16 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/01/22 12:27:16 INFO MemoryStore: MemoryStore cleared
16/01/22 12:27:16 INFO BlockManager: BlockManager stopped
16/01/22 12:27:16 INFO BlockManagerMaster: BlockManagerMaster stopped
16/01/22 12:27:16 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/01/22 12:27:16 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/01/22 12:27:16 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/01/22 12:27:16 INFO SparkContext: Successfully stopped SparkContext
16/01/22 12:27:16 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
Run completed in 58 seconds, 288 milliseconds.
Total number of tests run: 12
Suites: completed 4, aborted 0
Tests: succeeded 12, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org