如何使用U-SQL自定義提取器動態輸出多個文件？

Question

我創建了一個從多個文件中讀取數據的U-SQL查詢。現在我必須將數據輸出到其他文件夾中的多個文件中。我有下面的例子中，

U-SQL Output in Azure Data Lake

用下面的腳本，

 DECLARE @storagePath string = @"E:\"; 
     DECLARE @inputFileSetName string = @"dbo.file{*}.tsv"; 
     DECLARE @outputFileName string = @"dbo.files.csv"; 

     DECLARE @input string = String.Concat(@storagePath, 
     @inputFileSetName); 
     DECLARE @output = string.Concat(@storagePath, @outputFileName); 
     @searchlog = 
     EXTRACT Id string, 
     Name string, 
     Address string 

     FROM @input 
     USING new USQLApplication3.SampleExtractor(Encoding.UTF8); 
     @transactions = 
     SELECT *, 
     ROW_NUMBER() OVER(PARTITION BY Id ORDER BY Id DESC) AS 
     RowNumber_12345 
     FROM @searchlog; 

     @result = 
     SELECT EXTRACT Id, 
     Name, 
     Address 
     FROM @transactions 
     WHERE RowNumber_12345 == 1; 

     OUTPUT @result 
     TO @output 
     USING Outputters.Csv(encoding : Encoding.UTF8, quoting : false); 

下面是自定義提取器的C＃代碼，

public class SampleExtractor : IExtractor 
{ 
    private Encoding _encoding; 
    private byte[] _row_delim; 
    private char _col_delim; 

    public SampleExtractor(Encoding encoding, string row_delim = "\n", char col_delim = '\t') 
    { 
     this._encoding = ((encoding == null) ? Encoding.UTF8 : encoding); 
     this._row_delim = this._encoding.GetBytes(row_delim); 
     this._col_delim = col_delim; 
    } 

    public override IEnumerable<IRow> Extract(IUnstructuredReader input, IUpdatableRow output) 
    { 
     string line; 
     //Read the input line by line 
     foreach (Stream current in input.Split(_encoding.GetBytes("\n"))) 
     { 
      using (StreamReader streamReader = new StreamReader(current, this._encoding)) 
      { 
       line = streamReader.ReadToEnd().Trim(); 
       //Split the input by the column delimiter 
       string[] parts = line.Split(this._col_delim); 
       foreach (string part in parts) 
       { 
        string res = part; 
        if (res != null) 
        { 
         res = "\"" + res.Replace("\"", "\"\"") + "\""; 
        } 
        output.Set<string>(count, res); 
       } 

      } 
      yield return output.AsReadOnly(); 

     } 
     yield break; 
    } 
} 

哪有我動態創建多個文件輸出？截至目前，它只創建一個文件。

編輯：

我自己也嘗試在一個單獨的U-SQL腳本的多路輸出，

 @x = SELECT * FROM (VALUES("A", 10, 20), ("A", 11, 21), ("B", 10, 30), ("B", 100, 200)) AS T(name, value1, value2); 

    // Generate the script to do partitioned output based on name column: 
    DECLARE @out string ="E:/genscript.usql"; 
    @stmts = 
     SELECT "@res=SELECT value1, value2 FROM @x WHERE name == 
     \""+name+"\"; 
    OUTPUT 
      @res TO \"E:/"+name+".csv\" USING Outputters.Csv();" AS output 
      FROM (SELECT DISTINCT name FROM @x) AS x; 

    OUTPUT @stmts TO @out 
      USING Outputters.Text(delimiter:' ', quoting:false); 

    @path = EXTRACT path string FROM @out USING Extractors.Text() ; 
    OUTPUT @path TO "E:/{*}.usql" 
      USING Outputters.Text(delimiter:' ', quoting:false); 

卻是露出了下面的錯誤，

嚴重性代碼說明項目文件行抑制狀態 錯誤E_CSC_USER_READFROMPREVIOUSOUTPUT：EXTRACT語句從位置讀取/引用n E：\ genscript.usql，它是前一個OUTPUT語句的目標。 描述： 編譯器不支持OUTPUT，然後從單個作業中的相同文件讀取/引用。 分辨率： 請將一項工作輸出到文件，另一項工作是讀取/參考同一個文件。

我現在應該怎麼做，以輸出多個文件，從一個單獨的U-SQL或如何從我目前的u-SQL腳本訪問其他的u-SQL腳本文件？

Answer 1

你可以簡單地使用多個OUTPUT命令，例如

OUTPUT @result1 
TO @output1 
USING Outputters.Csv(encoding : Encoding.UTF8, quoting : false); 

OUTPUT @result2 
TO @output2 
USING Outputters.Csv(encoding : Encoding.UTF8, quoting : false); 
... 

U型SQL不支持動態U型SQL直接，但也有這樣做的方式。兩種示例方法是here和here。