Apache Spark Parquet：無法建立空的組

Question

我使用Apache Spark 2.1.1（使用2.1.0，它是相同的，今天切換）。 我有一個數據集：

root 
|-- muons: array (nullable = true) 
| |-- element: struct (containsNull = true) 
| | |-- reco::Candidate: struct (nullable = true) 
| | |-- qx3_: integer (nullable = true) 
| | |-- pt_: float (nullable = true) 
| | |-- eta_: float (nullable = true) 
| | |-- phi_: float (nullable = true) 
| | |-- mass_: float (nullable = true) 
| | |-- vertex_: struct (nullable = true) 
| | | |-- fCoordinates: struct (nullable = true) 
| | | | |-- fX: float (nullable = true) 
| | | | |-- fY: float (nullable = true) 
| | | | |-- fZ: float (nullable = true) 
| | |-- pdgId_: integer (nullable = true) 
| | |-- status_: integer (nullable = true) 
| | |-- cachePolarFixed_: struct (nullable = true) 
| | |-- cacheCartesianFixed_: struct (nullable = true) 

正如你所看到的，也有在此架構3層空結構。我知道100％我可以閱讀/操作/做任何事情。然而，當我嘗試在地板寫入磁盤，我得到以下異常：

dsReduced.write.format("parquet").save(outputPathName): 

java.lang.IllegalStateException: Cannot build an empty group 
at org.apache.parquet.Preconditions.checkState(Preconditions.java:91) 
at org.apache.parquet.schema.Types$BaseGroupBuilder.build(Types.java:622) 
at org.apache.parquet.schema.Types$BaseGroupBuilder.build(Types.java:497) 
at org.apache.parquet.schema.Types$Builder.named(Types.java:286) 
at org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter.convertField(ParquetSchemaConverter.scala:535) 
at org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter.convertField(ParquetSchemaConverter.scala:321) 
at org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter$$anonfun$convertField$1.apply(ParquetSchemaConverter.scala:534) 
at org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter$$anonfun$convertField$1.apply(ParquetSchemaConverter.scala:533) 

所以，基本上我想知道它是否是一個錯誤或預期的行爲???我也假設它與空洞的結構有關。任何幫助將非常感激！

更新：我已經快速創建了剝離版本，該工作沒有任何問題！任何見解都會非常有幫助！

VK

Answer 1

木地板不寫入空的結構：

更多信息 - 在這裏看到https://issues.apache.org/jira/browse/SPARK-20593

VK