提交彈簧啓動應用程序罐火花提交

Question

我很新的火花，我想嘗試火花提交。我在spring boot中創建了一個應用程序，使用mvn package來創建一個jar。但是，當我試圖提交罐子​​，它是無法找到主類。但是主要類是存在於罐子裏。

spark-submit --class com.dip.sparkapp.SparkappApplication --master local target/sparkapp-0.0.1-SNAPSHOT.jar 

Answer 1

我們遇到了同樣的問題，實際上，在您發佈此消息的同一天。我們的解決方案是使用Maven的陰影插件來編輯我們的構建。我們發現，當使用spring-boot-maven插件進行打包時，它將我們的類嵌套在BOOT-INF/classes類中，而spark類不喜歡這些類。我將粘貼相關部分，以便您可以在自己的應用程序中試用 - 祝您好運！

<build> 
    <plugins> 
     <plugin> 
      <groupId>org.apache.maven.plugins</groupId> 
      <artifactId>maven-shade-plugin</artifactId> 
      <dependencies> 
       <dependency> 
        <groupId>org.springframework.boot</groupId> 
        <artifactId>spring-boot-maven-plugin</artifactId> 
        <version>${spring-boot-version}</version> 
       </dependency> 
      </dependencies> 
      <configuration> 
       <keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope> 
       <createDependencyReducedPom>false</createDependencyReducedPom> 
       <filters> 
        <filter> 
         <artifact>*:*</artifact> 
         <excludes> 
          <exclude>META-INF/*.SF</exclude> 
          <exclude>META-INF/*.DSA</exclude> 
          <exclude>META-INF/*.RSA</exclude> 
         </excludes> 
        </filter> 
       </filters> 
       <transformers> 
        <transformer 
          implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> 
         <resource>META-INF/spring.handlers</resource> 
        </transformer> 
        <transformer 
          implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer"> 
         <resource>META-INF/spring.factories</resource> 
        </transformer> 
        <transformer 
          implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> 
         <resource>META-INF/spring.schemas</resource> 
        </transformer> 
        <transformer 
          implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" /> 
        <transformer 
          implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> 
         <mainClass>${start-class}</mainClass> 
        </transformer> 
       </transformers> 
      </configuration> 
      <executions> 
       <execution> 
        <phase>package</phase> 
        <goals> 
         <goal>shade</goal> 
        </goals> 
       </execution> 
      </executions> 
     </plugin> 
    </plugins> 
</build>