Windows上的Spark - 什麼是winutils，爲什麼我們需要它？

Question

我很好奇！據我所知，HDFS需要datanode進程才能運行，這就是爲什麼它只能在服務器上運行。 Spark雖然可以在本地運行，但需要winutils.exe，它是Hadoop的一個組件。但它究竟做了什麼？這是怎麼回事，我不能在Windows上運行Hadoop，但我可以運行基於Hadoop的Spark？

Answer 1

我知道至少有一個用法，它用於在Windows操作系統上運行shell命令。您可以在org.apache.hadoop.util.Shell找到它，其他的模塊依賴於這個類，並使用它的方法，例如getGetPermissionCommand()方法：

static final String WINUTILS_EXE = "winutils.exe"; 
... 
static { 
    IOException ioe = null; 
    String path = null; 
    File file = null; 
    // invariant: either there's a valid file and path, 
    // or there is a cached IO exception. 
    if (WINDOWS) { 
    try { 
     file = getQualifiedBin(WINUTILS_EXE); 
     path = file.getCanonicalPath(); 
     ioe = null; 
    } catch (IOException e) { 
     LOG.warn("Did not find {}: {}", WINUTILS_EXE, e); 
     // stack trace comes at debug level 
     LOG.debug("Failed to find " + WINUTILS_EXE, e); 
     file = null; 
     path = null; 
     ioe = e; 
    } 
    } else { 
    // on a non-windows system, the invariant is kept 
    // by adding an explicit exception. 
    ioe = new FileNotFoundException(E_NOT_A_WINDOWS_SYSTEM); 
    } 
    WINUTILS_PATH = path; 
    WINUTILS_FILE = file; 

    WINUTILS = path; 
    WINUTILS_FAILURE = ioe; 
} 
... 
public static String getWinUtilsPath() { 
    if (WINUTILS_FAILURE == null) { 
    return WINUTILS_PATH; 
    } else { 
    throw new RuntimeException(WINUTILS_FAILURE.toString(), 
     WINUTILS_FAILURE); 
    } 
} 
... 
public static String[] getGetPermissionCommand() { 
    return (WINDOWS) ? new String[] { getWinUtilsPath(), "ls", "-F" } 
        : new String[] { "/bin/ls", "-ld" }; 
}