Spark：是否有任何方法對大小寫不敏感的列進行排序（排序或orderBy）

Question

我需要以不區分大小寫的方式對行進行排序。

我有數據是這樣的：

+---+---------------+--------------------+--------------------+------+--------------+ 
| id|  full_name|   job_title|    email|gender| ip_address| 
+---+---------------+--------------------+--------------------+------+--------------+ 
| 73|  Tina Mccoy|Desktop Support T...|tmccoy20@techcrun...|Female| 23.196.170.54| 
| 74|  Lois Hart|  Food Chemist|lhart21@mapquest.com|Female| 145.52.30.236| 
| 75| Thomas Hall| Senior Developer| thall22@wired.com| Male|76.255.197.231| 
| 76| Ernest Romero|    Teacher|eromero23@amazon....| Male| 99.21.57.239| 
| 77| Irene Bradley| Assistant Professor|ibradley24@squido...|Female| 16.51.179.230| 
| 78|Jacqueline Cruz|account Represent...|  jcruz25@cdc.gov|Female| 167.49.98.213| 
| 79| Sara Martin|  Geologist IV| smartin26@a8.net|Female| 10.145.49.204| 
| 80| Johnny Bradley| Executive Secretary|jbradley27@cocolo...| Male| 138.251.4.102| 
| 81|  Fred Dean|Nuclear Power Eng...|fdean28@kickstart...| Male| 173.10.122.12| 
| 82| Ralph Greene|  Senior Editor|rgreene29@omnitur...| Male| 57.230.33.105| 
+---+---------------+--------------------+--------------------+------+--------------+ 

，當我使用df.orderBy('job_title')排序是基於job_title。這是我得到的。

+---+---------------+--------------------+--------------------+------+--------------+ 
| id|  full_name|   job_title|    email|gender| ip_address| 
+---+---------------+--------------------+--------------------+------+--------------+ 
| 77| Irene Bradley| Assistant Professor|ibradley24@squido...|Female| 16.51.179.230| 
| 73|  Tina Mccoy|Desktop Support T...|tmccoy20@techcrun...|Female| 23.196.170.54| 
| 80| Johnny Bradley| Executive Secretary|jbradley27@cocolo...| Male| 138.251.4.102| 
| 74|  Lois Hart|  Food Chemist|lhart21@mapquest.com|Female| 145.52.30.236| 
| 79| Sara Martin|  Geologist IV| smartin26@a8.net|Female| 10.145.49.204| 
| 81|  Fred Dean|Nuclear Power Eng...|fdean28@kickstart...| Male| 173.10.122.12| 
| 75| Thomas Hall| Senior Developer| thall22@wired.com| Male|76.255.197.231| 
| 82| Ralph Greene|  Senior Editor|rgreene29@omnitur...| Male| 57.230.33.105| 
| 76| Ernest Romero|    Teacher|eromero23@amazon....| Male| 99.21.57.239| 
| 78|Jacqueline Cruz|account Represent...|  jcruz25@cdc.gov|Female| 167.49.98.213| 
+---+---------------+--------------------+--------------------+------+--------------+ 

但我需要的是

+---+---------------+--------------------+--------------------+------+--------------+ 
| id|  full_name|   job_title|    email|gender| ip_address| 
+---+---------------+--------------------+--------------------+------+--------------+ 
| 78|Jacqueline Cruz|account Represent...|  jcruz25@cdc.gov|Female| 167.49.98.213| 
| 77| Irene Bradley| Assistant Professor|ibradley24@squido...|Female| 16.51.179.230| 
| 73|  Tina Mccoy|Desktop Support T...|tmccoy20@techcrun...|Female| 23.196.170.54| 
| 80| Johnny Bradley| Executive Secretary|jbradley27@cocolo...| Male| 138.251.4.102| 
| 74|  Lois Hart|  Food Chemist|lhart21@mapquest.com|Female| 145.52.30.236| 
| 79| Sara Martin|  Geologist IV| smartin26@a8.net|Female| 10.145.49.204| 
| 81|  Fred Dean|Nuclear Power Eng...|fdean28@kickstart...| Male| 173.10.122.12| 
| 75| Thomas Hall| Senior Developer| thall22@wired.com| Male|76.255.197.231| 
| 82| Ralph Greene|  Senior Editor|rgreene29@omnitur...| Male| 57.230.33.105| 
| 76| Ernest Romero|    Teacher|eromero23@amazon....| Male| 99.21.57.239| 
+---+---------------+--------------------+--------------------+------+--------------+ 

Answer 1

可以將計算表達式作爲參數傳遞給orderBy。因此你可以導入lower功能：

from pyspark.sql.functions import col, lower 

，並用它來包裝列名

df.orderBy(lower(col("job_title"))) 

Answer 2

簡單的解決方案將是該列創建job_title_lower_case欄然後進行排序。在最終結果中，只需放下新的列即可。