如何在Scala和Apache Spark中連接兩個DataFrame？

有兩個DataFrames（Scala中，阿帕奇星火1.6.1）如何在Scala和Apache Spark中連接兩個DataFrame？

1）匹配

  MatchID | Player1 | Player2 
     -------------------------------- 
       1 | John Wayne | John Doe 
       2 | Ive Fish | San Simon

2）個人資料

   Player  | BirthYear 
       -------------------------------- 
       John Wayne | 1986 
       Ive Fish | 1990 
       San Simon | 1974 
       john Doe | 1995

怎麼會用 'BirthYear' 一個新的數據幀對於雙方球員來說

  MatchID | Player1 | Player2 | BYear_P1 |BYear_P2 | Diff 
     ------------------------------------------------------------- 
       1 | John Wayne | John Doe | 1986 | 1995 | 9 
       2 | Ive Fish | San Simon | 1990 | 1974 | 16

？

我試圖

val df = MatchesDF.join(PersonalDF, MatchesDF("Player1") === PersonalDF("Player"))

然後第二個玩家

val resDf = df.join(PersonalDF, df("Player2") === PersonalDF("Player"))

再次加入，但它是非常耗時的操作。

可能是Scala和Apache Spark中的另一種方法嗎？

來源

2016-04-22 gmlvsv

這應該有更好的表現：

case class Match(matchId: Int, player1: String, player2: String) 
case class Player(name: String, birthYear: Int) 

val matches = Seq(
    Match(1, "John Wayne", "John Doe"), 
    Match(2, "Ive Fish", "San Simon") 
) 

val players = Seq(
    Player("John Wayne", 1986), 
    Player("Ive Fish", 1990), 
    Player("San Simon", 1974), 
    Player("John Doe", 1995) 
) 

val matchesDf = sqlContext.createDataFrame(matches) 
val playersDf = sqlContext.createDataFrame(players) 

matchesDf.registerTempTable("matches") 
playersDf.registerTempTable("players") 

sqlContext.sql(
    "select matchId, player1, player2, p1.birthYear, p2.birthYear, abs(p1.birthYear-p2.birthYear) " + 
    "from matches m inner join players p1 inner join players p2 " + 
    "where m.player1 = p1.name and m.player2 = p2.name").show() 

+-------+----------+---------+---------+---------+---+ 
|matchId| player1| player2|birthYear|birthYear|_c5| 
+-------+----------+---------+---------+---------+---+ 
|  1|John Wayne| John Doe|  1986|  1995| 9| 
|  2| Ive Fish|San Simon|  1990|  1974| 16| 
+-------+----------+---------+---------+---------+---+

我沒有找到表達加入斯卡拉DSL 3代表的方式。

來源

2016-04-23 07:46:49

再次做兩聯接，它是如何做的更好？ – void

這對匹配表運行約2分鐘，約10000行，玩家表約700記錄 – gmlvsv

使用數據框爲您的連接，而不是普通的SQL更好的性能。 – dheee

val df = left.join(right, Seq("name")) 
display(df)

來源

2017-01-26 23:20:43

你好，歡迎來到StackOverflow。請爲您的答案添加一些解釋，以便對其他用戶更有價值。請參閱http://stackoverflow.com/help/how-to-answer – wmk

此信息不足以提供任何形式的幫助。剩下什麼」？什麼是對的」？請更改您的答案。 –

Spark Dataframe中沒有顯示功能（Scala實現） –

這是一個解決方案使用火花的數據幀功能：

import sqlContext.implicits._ 
import org.apache.spark.sql.Row 
import org.apache.spark.sql.functions.abs 

val matches = sqlContext.sparkContext.parallelize(Row(1, "John Wayne", "John Doe"), Row(2, "Ive Fish", "San Simon"))) 

val players = sqlContext.sparkContext.parallelize(Seq(
    Row("John Wayne", 1986), 
    Row("Ive Fish", 1990), 
    Row("San Simon", 1974), 
    Row("John Doe", 1995) 
)) 

val matchesDf = sqlContext.createDataFrame(matches, StructType(Seq(
    StructField("matchId", IntegerType, nullable = false), 
    StructField("player1", StringType, nullable = false), 
    StructField("player2", StringType, nullable = false))) 
).as('matches) 

val playersDf = sqlContext.createDataFrame(players, StructType(Seq(
    StructField("player", StringType, nullable = false), 
    StructField("birthYear", IntegerType, nullable = false) 
))).as('players) 

matchesDf 
    .join(playersDf, $"matches.player1" === $"players.player") 
    .select($"matches.matchId" as "matchId", $"matches.player1" as "player1", $"matches.player2" as "player2", $"players.birthYear" as "player1BirthYear") 
    .join(playersDf, $"player2" === $"players.player") 
    .select($"matchId" as "MatchID", $"player1" as "Player1", $"player2" as "Player2", $"player1BirthYear" as "BYear_P1", $"players.birthYear" as "BYear_P2") 
    .withColumn("Diff", abs('BYear_P2.minus('BYear_P1))) 
    .show() 

+-------+----------+---------+--------+--------+----+ 
|MatchID| Player1| Player2|BYear_P1|BYear_P2|Diff| 
+-------+----------+---------+--------+--------+----+ 
|  1|John Wayne| John Doe| 1986| 1995| 9| 
|  2| Ive Fish|San Simon| 1990| 1974| 16| 
+-------+----------+---------+--------+--------+----+

來源

2017-08-18 04:16:14

如何在Scala和Apache Spark中連接兩個DataFrame？

回答

相關問題