U-SQL根據文件中的Distinct值將CSV文件拆分爲多個文件

Question

我在Azure Data Lake Store中有數據，我正在使用U-SQL處理Azure Data Analytic Job中存在的數據。我有包含空間數據的幾個CSV文件，與此類似：

File_20170301.csv

longtitude| lattitude | date   | hour | value1 
    ----------+-----------+--------------+------+------- 
    45.121 | 21.123 | 2017-03-01 | 01 | 20 
    45.121 | 21.123 | 2017-03-01 | 02 | 10 
    45.121 | 21.123 | 2017-03-01 | 03 | 50 
    48.121 | 35.123 | 2017-03-01 | 01 | 60 
    48.121 | 35.123 | 2017-03-01 | 02 | 15 
    48.121 | 35.123 | 2017-03-01 | 03 | 80 

File_20170302.csv

longtitude| lattitude | date   | hour | value1 
    ----------+-----------+--------------+------+------- 
    45.121 | 21.123 | 2017-03-02 | 01 | 20 
    45.121 | 21.123 | 2017-03-02 | 02 | 10 
    45.121 | 21.123 | 2017-03-02 | 03 | 50 
    48.121 | 35.123 | 2017-03-02 | 01 | 60 
    48.121 | 35.123 | 2017-03-02 | 02 | 15 
    48.121 | 35.123 | 2017-03-02 | 03 | 80 

每個文件包含的數據不同的日期和所有經度 - 組合。

我想合併我所有的文件並拆分數據，這樣我就可以爲每個經度 - 折點組合結束一個文件。

所以，通過我的文件夾中的所有文件循環和附加的所有數據全部天后，我將結束與以下：

File_45_21.csv

longtitude| lattitude | date   | hour | value1 
    ----------+-----------+--------------+------+------- 
    45.121 | 21.123 | 2017-03-01 | 01 | 20 
    45.121 | 21.123 | 2017-03-01 | 02 | 10 
    45.121 | 21.123 | 2017-03-01 | 03 | 50 
    45.121 | 21.123 | 2017-03-02 | 01 | 20 
    45.121 | 21.123 | 2017-03-02 | 02 | 10 
    45.121 | 21.123 | 2017-03-02 | 03 | 50 

File_48_35.csv

longtitude| lattitude | date   | hour | value1 
    ----------+-----------+--------------+------+------- 
    48.121 | 35.123 | 2017-03-01 | 01 | 60 
    48.121 | 35.123 | 2017-03-01 | 02 | 15 
    48.121 | 35.123 | 2017-03-01 | 03 | 80 
    48.121 | 35.123 | 2017-03-02 | 01 | 60 
    48.121 | 35.123 | 2017-03-02 | 02 | 15 
    48.121 | 35.123 | 2017-03-02 | 03 | 80 

理論上下面應該發生：

1. 查找數據中的經度和緯度組合的不同值
2. 取上述不同值的數組，併爲每個組合創建一個文件，並根據兩個參數從源文件中提取數據（長度和012）

我正在努力的是如何開始循環和基於源中的兩個參數提取數據，以及如何通過組合的不同值對數據源進行「分區」的參數。

Answer 1

U-SQL不直接支持動態U-SQL，但可以使用「腳本腳本」技術來創建輸出。然後，您可以手動運行此輸出，或者使用Powershell或Azure Data Factory等軟件來運行它。

我已經根據您的測試數據部分基於here的示例創建了一個簡單示例。

// Get the initial fileset 
@input = 
    EXTRACT longtitude float, 
      lattitude float, 
      date string, 
      hour int, 
      value1 int, 
      filename string 
    FROM "/input/File_201703{filename}" 
    USING Extractors.Csv(); 


// Add int version of the long and lat columns for grouping on 
@working = 
    SELECT *, 
      (int) longtitude AS int_long, 
      (int) lattitude AS int_lat 
    FROM @input; 


// Work out the filenames 
@filenames = 
    SELECT String.Format("File {0}_{1}.csv", int_long, int_lat) AS outputFilename, 
      int_long, 
      int_lat 
    FROM 
    (
     SELECT int_long, 
       int_lat 
     FROM @working 
     GROUP BY int_long, 
       int_lat 
    ) AS x; 


// Construct the dynamic usql and output it 
@output = 
    SELECT x.xsort, "@input = EXTRACT longtitude float, lattitude float, date string, hour int, value1 int, filename string FROM \"input/File_201703{filename}\" USING Extractors.Csv();" AS usql 
    FROM (VALUES (10)) AS x(xsort) 

    UNION ALL 

    SELECT x.xsort, "@working = SELECT *, (int) longtitude AS int_long, (int) lattitude AS int_lat FROM @input;" AS usql 
    FROM (VALUES (20)) AS x(xsort) 

    UNION ALL 

    SELECT 30 AS xsort, String.Format("OUTPUT (SELECT * FROM @working WHERE int_long == {0} AND int_lat == {1}) TO \"/output/{2}\" USING Outputters.Csv();", int_long, int_lat, outputFilename) AS usql 
    FROM @filenames; 


// Select only the usql column and sort the output 
@output = 
    SELECT usql 
    FROM @output 
ORDER BY xsort 
FETCH 100; 


OUTPUT @output 
TO "/output/dynamic.usql" 
USING Outputters.Text(delimiter : ' ', quoting : false); 

Answer 2

我會建議看文件的概念設定到許多文件（見https://msdn.microsoft.com/en-us/library/azure/mt771650.aspx）和一些建議操作以做基於價值的動態輸出，直到該功能可用（見How do I partition a large file into files/directories using only U-SQL and certain fields in the file?作爲一個例子）。