2017-08-27 24 views
1

我想在Spark RDD使用HList從無形的,我想用toList方法,需要以下隱:無形ToTraversable不serialisable

implicit ev2: ToTraversable.Aux[InType, List, OutType] 

然而,如果我通過這個隱含的在RDD,它拋出一個NotSerializableException

Caused by: java.io.NotSerializableException: scala.collection.generic.GenTraversableFactory$$anon$1 
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) 
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) 
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) 
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 

是否有其他方式來實現在無形toList,還是有辦法提供一個隱含的是serialisable?

回答

1

閱讀throught一些無形的代碼後,我在tests發現,解決方法是將這種隱含CanBuildFrom範圍爲ToTraversable

implicit def listSerializableCanBuildFrom[T]: CanBuildFrom[List[T], T, List[T]] = 
    new CanBuildFrom[List[T], T, List[T]] with Serializable { 
     def apply(from: List[T]) = from.genericBuilder[T] 

     def apply() = List.newBuilder[T] 
    }