2017-05-02 78 views
1

我有一個maven構建,我在Docker容器中使用Docker中心的官方Maven圖像在Docker容器中運行。 .m2目錄被掛載到一個NFS共享中。Maven可以創建文件夾和鎖定文件,但下載時掛起

這可以在一個環境中使用,但在另一個相同的環境中,它在寫入鎖定文件後將始終掛起。它從來沒有完成下載,但永遠掛在那裏。由於maven debug在掛起點後沒有提供任何細節,因此我決定觀看.m2目錄以查看發生了什麼。

[email protected]:/nfs-shares/jenkins/.m2$ inotifywait -m -r . 
Setting up watches. Beware: since -r was given, this may take a while! 
Watches established. 
./ CREATE,ISDIR repository 
./ OPEN,ISDIR repository 
./ CLOSE_NOWRITE,CLOSE,ISDIR repository 
./repository/ CREATE,ISDIR org 
./repository/ OPEN,ISDIR org 
./repository/ CLOSE_NOWRITE,CLOSE,ISDIR org 
./repository/org/ CREATE,ISDIR springframework 
./repository/org/ OPEN,ISDIR springframework 
./repository/org/ CLOSE_NOWRITE,CLOSE,ISDIR springframework 
./repository/org/springframework/ CREATE,ISDIR boot 
./repository/org/springframework/ OPEN,ISDIR boot 
./repository/org/springframework/ CLOSE_NOWRITE,CLOSE,ISDIR boot 
./repository/org/springframework/boot/ CREATE,ISDIR spring-boot-starter-parent 
./repository/org/springframework/boot/ OPEN,ISDIR spring-boot-starter-parent 
./repository/org/springframework/boot/ CLOSE_NOWRITE,CLOSE,ISDIR spring-boot-starter-parent 
./repository/org/springframework/boot/spring-boot-starter-parent/ CREATE,ISDIR 1.3.7.RELEASE 
./repository/org/springframework/boot/spring-boot-starter-parent/ OPEN,ISDIR 1.3.7.RELEASE 
./repository/org/springframework/boot/spring-boot-starter-parent/ CLOSE_NOWRITE,CLOSE,ISDIR 1.3.7.RELEASE 
./repository/org/springframework/boot/spring-boot-starter-parent/1.3.7.RELEASE/ CREATE spring-boot-starter-parent-1.3.7.RELEASE.pom.part.lock 

Maven似乎工作,它創建了一些文件夾,甚至鎖定文件,但它然後掛起。我如何獲得maven來完成或找到一些額外的信息來幫助我解決這個問題。

順便說一句,如果我在容器內使用臨時存儲,它會按預期下載軟件包。

UDPATE:其中一條評論提示了一個線程轉儲。在下面你可以看到我連接到正在運行的容器。我確認容器可以修改.m2目錄中的文件,然後使用jstack獲取進程的線程轉儲。

[email protected]:~$ sudo docker ps 
CONTAINER ID  IMAGE              COMMAND     CREATED    STATUS    PORTS    NAMES 
c7d1f4c91559  maven:alpine            "cat"     About an hour ago Up About an hour      agitated_cori 
[email protected]:~$ sudo docker exec -ti c7d1f4c91559 /bin/bash 
bash-4.3$ ps 
PID USER  TIME COMMAND 
    1 1000  0:00 cat 
    6 1000  0:00 sh -c echo $$ > '/var/jenkins_home/workspace/[email protected]/durable-ca9825bd/pid'; jsc=durable-04ba6b757bca34373f180bd01ef64ca1; JENKINS_SERVER_COOKIE=$jsc '/var/jenkins_home/workspace/[email protected]/durable-ca 
    12 1000  0:00 {script.sh} /bin/sh -xe /var/jenkins_home/workspace/[email protected]/durable-ca9825bd/script.sh 
    13 1000  0:07 /usr/lib/jvm/java-1.8-openjdk/bin/java -classpath /usr/share/maven/boot/plexus-classworlds-2.5.2.jar -Dclassworlds.conf=/usr/share/maven/bin/m2.conf -Dmaven.home=/usr/share/maven -Dmaven.multiModuleProjectDirecto 
1584 1000  0:00 /bin/bash 
1589 1000  0:00 ps 
bash-4.3$ cat /var/jenkins_home/workspace/[email protected]/durable-ca9825bd/script.sh 
#!/bin/sh -xe 
mvn -Dmaven.repo.local="$PWD"/../../.m2/repository clean compile 
bash-4.3$ ls -la /var/jenkins_home/.m2/ 
total 16 
drwxr-xr-x 3 1000  1000   4096 May 2 21:14 . 
drwxrwxr-x 23 1000  1000   4096 May 3 11:55 .. 
-rw-r--r-- 1 1000  1000    6 May 2 21:14 file.txt 
drwxr-xr-x 3 1000  1000   4096 May 2 20:50 repository 
bash-4.3$ cat /var/jenkins_home/.m2/file.txt 
hello 
bash-4.3$ vi /var/jenkins_home/.m2/file.txt 
bash-4.3$ cat /var/jenkins_home/.m2/file.txt 
hello 
another 

bash-4.3$ jstack 13 
2017-05-03 13:04:37 
Full thread dump OpenJDK 64-Bit Server VM (25.121-b13 mixed mode): 

"Attach Listener" #11 daemon prio=9 os_prio=0 tid=0x00007fc4a4956800 nid=0x6a7 runnable [0x0000000000000000] 
    java.lang.Thread.State: RUNNABLE 

"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fc4a4343000 nid=0x2c runnable [0x0000000000000000] 
    java.lang.Thread.State: RUNNABLE 

"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fc4a4311800 nid=0x2b waiting on condition [0x0000000000000000] 
    java.lang.Thread.State: RUNNABLE 

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fc4a4302000 nid=0x2a waiting on condition [0x0000000000000000] 
    java.lang.Thread.State: RUNNABLE 

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fc4a42ff000 nid=0x29 waiting on condition [0x0000000000000000] 
    java.lang.Thread.State: RUNNABLE 

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fc4a42fc800 nid=0x28 runnable [0x0000000000000000] 
    java.lang.Thread.State: RUNNABLE 

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fc4a42d5000 nid=0x27 in Object.wait() [0x00007fc48ba4b000] 
    java.lang.Thread.State: WAITING (on object monitor) 
    at java.lang.Object.wait(Native Method) 
    - waiting on <0x00000000dab108d8> (a java.lang.ref.ReferenceQueue$Lock) 
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) 
    - locked <0x00000000dab108d8> (a java.lang.ref.ReferenceQueue$Lock) 
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) 
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) 

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fc4a42ca800 nid=0x26 in Object.wait() [0x00007fc48bb4c000] 
    java.lang.Thread.State: WAITING (on object monitor) 
    at java.lang.Object.wait(Native Method) 
    - waiting on <0x00000000dab18178> (a java.lang.ref.Reference$Lock) 
    at java.lang.Object.wait(Object.java:502) 
    at java.lang.ref.Reference.tryHandlePending(Reference.java:191) 
    - locked <0x00000000dab18178> (a java.lang.ref.Reference$Lock) 
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) 

"main" #1 prio=5 os_prio=0 tid=0x00007fc4a4179800 nid=0x20 runnable [0x00007fc4a3426000] 
    java.lang.Thread.State: RUNNABLE 
    at sun.nio.ch.FileDispatcherImpl.lock0(Native Method) 
    at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:90) 
    at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1115) 
    at org.eclipse.aether.connector.basic.PartialFile$LockFile.tryLock(PartialFile.java:135) 
    at org.eclipse.aether.connector.basic.PartialFile$LockFile.lock(PartialFile.java:80) 
    at org.eclipse.aether.connector.basic.PartialFile$LockFile.<init>(PartialFile.java:67) 
    at org.eclipse.aether.connector.basic.PartialFile$Factory.newInstance(PartialFile.java:219) 
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:441) 
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:359) 
    at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:76) 
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:590) 
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:258) 
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:529) 
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:430) 
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:255) 
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:232) 
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:303) 
    at org.apache.maven.project.ProjectModelResolver.resolveModel(ProjectModelResolver.java:193) 
    at org.apache.maven.project.ProjectModelResolver.resolveModel(ProjectModelResolver.java:243) 
    at org.apache.maven.model.building.DefaultModelBuilder.readParentExternally(DefaultModelBuilder.java:1051) 
    at org.apache.maven.model.building.DefaultModelBuilder.readParent(DefaultModelBuilder.java:829) 
    at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:331) 
    at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:429) 
    at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:398) 
    at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:361) 
    at org.apache.maven.graph.DefaultGraphBuilder.collectProjects(DefaultGraphBuilder.java:400) 
    at org.apache.maven.graph.DefaultGraphBuilder.getProjectsForMavenReactor(DefaultGraphBuilder.java:391) 
    at org.apache.maven.graph.DefaultGraphBuilder.build(DefaultGraphBuilder.java:78) 
    at org.apache.maven.DefaultMaven.buildGraph(DefaultMaven.java:511) 
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:221) 
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194) 
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107) 
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993) 
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345) 
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:191) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) 
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) 
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) 
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) 

"VM Thread" os_prio=0 tid=0x00007fc4a42c0000 nid=0x25 runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fc4a4190800 nid=0x21 runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fc4a4192000 nid=0x22 runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fc4a4194000 nid=0x23 runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fc4a4195800 nid=0x24 runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007fc4a4382800 nid=0x2d waiting on condition 

JNI global references: 235 

bash-4.3$ 

容器我裏面只是證實,我可以訪問POM和我顯示調試輸出,剛剛掛起,在下載。

https://gist.github.com/dwatrous/34e1edc1db5e4756d4b33c83a9c2ccd0

+0

我只是確認,我可以從Maven容器的.m2目錄目錄中寫入文件,所以我真的懷疑這與NFS或文件權限有關。這似乎主要是一個maven問題。 –

+0

當不使用NFS時它工作嗎?因爲它在另一個環境中工作,所以必須有所不同。 –

+0

掛起的線程轉儲對診斷問題很有用。 –

回答

1

極有可能涉及到NFS和文件鎖定的錯誤和/或語義。

其他人通過NFS報告了FileChannel#tryLock的類似問題;見例如JDK-8156026JDK-8065927

contract of that method saystryLock不會阻塞,所以發生的任何阻塞都是由於本機系統調用不應返回。 Maven可能會試圖繞過這些bug,但我認爲任何嘗試這樣做都會很冒險,並且可能會引入更多的錯誤而不是避免。

你可以嘗試不同的Java版本,包括Oracle和OpenJDK的,在不同的發行版本......

+0

我懷疑這個答案是正確的方向,但我不知道如何進一步排除故障。在兩種環境中,一種工作,另一種不工作。我很想弄清楚這個NFS服務器有什麼問題,所以它可以像另一個一樣工作。我沒有看到你們聯繫的兩個問題中提到的決議。 –

+0

你說你的兩個環境是「相同的」,但_something_必須不同,不是嗎?是否在一個系統上確定性地重現掛起,並且確定性地不存在於另一個系統上?你是否在多個系統上同時運行構建?我當然不是NFS專家,但也許這篇文章有用:https://docstore.mik.ua/orelly/networking_2ndEd/nfs/ch11_03.htm? – ctrueden

+0

我在看的一個區別是,工作系統使用臨時磁盤,而掛在文件鎖的系統使用cinder捲來存儲NFS數據。 –

相關問題