如何在MATLAB中加載MNIST數字和標籤數據？

Question

我試圖運行中的鏈接

https://github.com/bd622/DiscretHashing

離散哈希給出的代碼是降維的方法是在近似最近鄰搜索中使用。我想加載在http://yann.lecun.com/exdb/mnist/中可用的MNIST數據庫上的實現。我已經從壓縮的gz格式中提取文件。

問題1：

利用該解決方案來讀取MNIST數據庫Reading MNIST Image Database binary file in MATLAB

提供我收到以下錯誤：

Error using fread 
Invalid file identifier. Use fopen to generate a valid file identifier. 

Error in Reading (line 7) 
A = fread(fid, 1, 'uint32'); 

下面是代碼：

clear all; 
close all; 

%//Open file 
fid = fopen('t10k-images-idx3-ubyte', 'r'); 

A = fread(fid, 1, 'uint32'); 
magicNumber = swapbytes(uint32(A)); 

%//For each image, store into an individual cell 
imageCellArray = cell(1, totalImages); 
for k = 1 : totalImages 
    %//Read in numRows*numCols pixels at a time 
    A = fread(fid, numRows*numCols, 'uint8'); 
    %//Reshape so that it becomes a matrix 
    %//We are actually reading this in column major format 
    %//so we need to transpose this at the end 
    imageCellArray{k} = reshape(uint8(A), numCols, numRows)'; 
end 

%//Close the file 
fclose(fid); 

UPDATE：問題1解決，並且修改後的代碼是

clear all; 
close all; 

%//Open file 
fid = fopen('t10k-images.idx3-ubyte', 'r'); 

A = fread(fid, 1, 'uint32'); 
magicNumber = swapbytes(uint32(A)); 

%//Read in total number of images 
%//A = fread(fid, 4, 'uint8'); 
%//totalImages = sum(bitshift(A', [24 16 8 0])); 

%//OR 
A = fread(fid, 1, 'uint32'); 
totalImages = swapbytes(uint32(A)); 

%//Read in number of rows 
%//A = fread(fid, 4, 'uint8'); 
%//numRows = sum(bitshift(A', [24 16 8 0])); 

%//OR 
A = fread(fid, 1, 'uint32'); 
numRows = swapbytes(uint32(A)); 

%//Read in number of columns 
%//A = fread(fid, 4, 'uint8'); 
%//numCols = sum(bitshift(A', [24 16 8 0])); 

%// OR 
A = fread(fid, 1, 'uint32'); 
numCols = swapbytes(uint32(A)); 

for k = 1 : totalImages 
    %//Read in numRows*numCols pixels at a time 
    A = fread(fid, numRows*numCols, 'uint8'); 
    %//Reshape so that it becomes a matrix 
    %//We are actually reading this in column major format 
    %//so we need to transpose this at the end 
    imageCellArray{k} = reshape(uint8(A), numCols, numRows)'; 
end 

%//Close the file 
fclose(fid); 

問題2：

我無法理解如何將4個文件的MNIST應用中的代碼。代碼包含變量

traindata = double(traindata); 
testdata = double(testdata); 

如何準備MNIST數據庫以便我可以應用於實施？

UPDATE：我實現瞭解決方案，但我不斷收到此錯誤

Error using fread 
Invalid file identifier. Use fopen to generate a valid file identifier. 

Error in mnist_parse (line 11) 
A = fread(fid1, 1, 'uint32'); 

這些文件

demo.m％，這是調用該函數在MNIST數據讀取主文件

clear all 
clc 
[Trainimages, Trainlabels] = mnist_parse('C:\Users\Desktop\MNIST\train-images-idx3-ubyte', 'C:\Users\Desktop\MNIST\train-labels-idx1-ubyte'); 

[Testimages, Testlabels] = mnist_parse('t10k-images-idx3-ubyte', 't10k-labels-idx1-ubyte'); 

k=5; 
digit = images(:,:,k); 
lbl = label(k); 

---

function [images, labels] = mnist_parse(path_to_digits, path_to_labels) 

% Open files 
fid1 = fopen(path_to_digits, 'r'); 

% The labels file 
fid2 = fopen(path_to_labels, 'r'); 

% Read in magic numbers for both files 
A = fread(fid1, 1, 'uint32'); 
magicNumber1 = swapbytes(uint32(A)); % Should be 2051 
fprintf('Magic Number - Images: %d\n', magicNumber1); 

A = fread(fid2, 1, 'uint32'); 
magicNumber2 = swapbytes(uint32(A)); % Should be 2049 
fprintf('Magic Number - Labels: %d\n', magicNumber2); 

% Read in total number of images 
% Ensure that this number matches with the labels file 
A = fread(fid1, 1, 'uint32'); 
totalImages = swapbytes(uint32(A)); 
A = fread(fid2, 1, 'uint32'); 
if totalImages ~= swapbytes(uint32(A)) 
    error('Total number of images read from images and labels files are not the same'); 
end 
fprintf('Total number of images: %d\n', totalImages); 

% Read in number of rows 
A = fread(fid1, 1, 'uint32'); 
numRows = swapbytes(uint32(A)); 

% Read in number of columns 
A = fread(fid1, 1, 'uint32'); 
numCols = swapbytes(uint32(A)); 

fprintf('Dimensions of each digit: %d x %d\n', numRows, numCols); 

% For each image, store into an individual slice 
images = zeros(numRows, numCols, totalImages, 'uint8'); 
for k = 1 : totalImages 
    % Read in numRows*numCols pixels at a time 
    A = fread(fid1, numRows*numCols, 'uint8'); 

    % Reshape so that it becomes a matrix 
    % We are actually reading this in column major format 
    % so we need to transpose this at the end 
    images(:,:,k) = reshape(uint8(A), numCols, numRows).'; 
end 

% Read in the labels 
labels = fread(fid2, totalImages, 'uint8'); 

% Close the files 
fclose(fid1); 
fclose(fid2); 

end 

Answer 1

我是方法＃1的原始作者，您提到過。讀取培訓數據和測試標籤的過程非常簡單。就讀取圖像而言，上面顯示的代碼完美地讀取文件，並且處於單元格陣列格式。但是，您在中缺少讀取圖像數量，行數和列數爲的文件。請注意，該文件的MNIST格式採用以下方式。左邊一列是你相對於參考的開始字節偏移量：

[offset] [type]   [value]   [description] 
0000  32 bit integer 0x00000803(2051) magic number 
0004  32 bit integer 60000   number of images 
0008  32 bit integer 28    number of rows 
0012  32 bit integer 28    number of columns 
0016  unsigned byte ??    pixel 
0017  unsigned byte ??    pixel 
........ 
xxxx  unsigned byte ??    pixel 

前四個字節是一個神奇的數字：2051，以確保您的文件中正確讀取。接下來的四個字節表示圖像的總數，接下來的四個字節是行，最後四個字節是列。應該有2800×28×28的大小的圖像。在此之後，像素以行主格式交織，因此您必須循環顯示一系列28 x 28像素並存儲它們。在這種情況下，我將它們存儲在單元格數組中，並且此單元格數組中的每個元素都是一個數字。測試數據的格式也相同，但是有10000個圖像。

至於實際的標籤，這是大致相同的格式，但也有一些細微的差別：

[offset] [type]   [value]   [description] 
0000  32 bit integer 0x00000801(2049) magic number (MSB first) 
0004  32 bit integer 60000   number of items 
0008  unsigned byte ??    label 
0009  unsigned byte ??    label 
........ 
xxxx  unsigned byte ??    label 

前四個字節是一個神奇的數字：2049，然後是第二組的四個字節告訴您如何那裏有很多標籤，最後數據集中每個對應的數字都有1個字節。測試數據也是相同的格式，但有10000個標籤。因此，一旦讀入標籤集中的必要數據，您只需要一個fread調用，並確保數據是無符號的8位整數，以讀取其餘標籤。

現在你爲什麼要使用swapbytes的原因是因爲MATLAB將在小端格式的數據讀取，這意味着從一組字節至少顯著字節在第一個被讀取。完成後，您可以使用swapbytes重新排列此順序。

因此，我已經修改了這段代碼，以便它是一個實際的函數，它接受一組兩個字符串：數字圖像文件的完整路徑和數字的完整路徑。我還更改了代碼，以便圖像是3D數字矩陣，而不是單元陣列，以便進行更快的處理。請注意，當您開始讀取實際圖像數據時，每個像素都是無符號的8位整數，因此不需要進行任何字節交換。

function [images, labels] = mnist_parse(path_to_digits, path_to_labels) 

% Open files 
fid1 = fopen(path_to_digits, 'r'); 

% The labels file 
fid2 = fopen(path_to_labels, 'r'); 

% Read in magic numbers for both files 
A = fread(fid1, 1, 'uint32'); 
magicNumber1 = swapbytes(uint32(A)); % Should be 2051 
fprintf('Magic Number - Images: %d\n', magicNumber1); 

A = fread(fid2, 1, 'uint32'); 
magicNumber2 = swapbytes(uint32(A)); % Should be 2049 
fprintf('Magic Number - Labels: %d\n', magicNumber2); 

% Read in total number of images 
% Ensure that this number matches with the labels file 
A = fread(fid1, 1, 'uint32'); 
totalImages = swapbytes(uint32(A)); 
A = fread(fid2, 1, 'uint32'); 
if totalImages ~= swapbytes(uint32(A)) 
    error('Total number of images read from images and labels files are not the same'); 
end 
fprintf('Total number of images: %d\n', totalImages); 

% Read in number of rows 
A = fread(fid1, 1, 'uint32'); 
numRows = swapbytes(uint32(A)); 

% Read in number of columns 
A = fread(fid1, 1, 'uint32'); 
numCols = swapbytes(uint32(A)); 

fprintf('Dimensions of each digit: %d x %d\n', numRows, numCols); 

% For each image, store into an individual slice 
images = zeros(numRows, numCols, totalImages, 'uint8'); 
for k = 1 : totalImages 
    % Read in numRows*numCols pixels at a time 
    A = fread(fid1, numRows*numCols, 'uint8'); 

    % Reshape so that it becomes a matrix 
    % We are actually reading this in column major format 
    % so we need to transpose this at the end 
    images(:,:,k) = reshape(uint8(A), numCols, numRows).'; 
end 

% Read in the labels 
labels = fread(fid2, totalImages, 'uint8'); 

% Close the files 
fclose(fid1); 
fclose(fid2); 

end 

調用這個函數，只需指定兩個圖像文件和標籤文件的路徑：這多個字節讀一個fread電話時，只要求。假設你正在運行在該文件所在的同一目錄下這個文件，你會做的訓練圖像執行下列操作之一：

[images, labels] = mnist_parse('train-images-idx3-ubyte', 'train-labels-idx1-ubyte'); 

而且，你會做以下的測試圖像：

[images, labels] = mnist_parse('t10k-images-idx3-ubyte', 't10k-labels-idx1-ubyte'); 

要訪問k個數字，你會簡單地做：

digit = images(:,:,k); 

爲k個位數對應的標籤是：

lbl = label(k); 

---

終於得到這個數據到一個格式，是，我已經看到在Github上的代碼可以接受的，他們認爲該行對應於訓練樣本和列對應的功能。如果你想要這種格式，只需重塑數據，以便圖像像素分佈在列上。

因此，只是這樣做：

[trainingdata, traingnd] = mnist_parse('train-images-idx3-ubyte', 'train-labels-idx1-ubyte'); 
trainingdata = double(reshape(trainingdata, size(trainingdata,1)*size(trainingdata,2), []).'); 
traingnd = double(traingnd); 

[testdata, testgnd] = mnist_parse('t10k-images-idx3-ubyte', 't10k-labels-idx1-ubyte'); 
testdata = double(reshape(testdata, size(testdata,1)*size(testdata_data,2), []).'); 
testgnd = double(testgnd); 

以上使用相同的變量在腳本，所以你應該能夠在堵塞這一點，它應該工作。第二行重塑矩陣，以便每個數字位於列中，但我們需要對此進行轉置，以便每個數字位於列中。我們還需要將其轉換爲double，因爲這是Github代碼的功能。相同的邏輯適用於測試數據。另外請注意，我已明確將培訓和測試標籤投放到double，以確保您決定使用此數據的任何算法具有最大的兼容性。

---

快樂的數字黑客！