如何使用scipy的affine_transform對彩色圖像進行任意仿射變換？

Question

我的目標是以這樣一種方式轉換圖像，即將三個源點映射到空數組中的三個目標點。我已經解決了正確的仿射矩陣的發現，但是我無法對彩色圖像應用仿射變換。

更具體地說，我正在努力正確使用scipy.ndimage.interpolation.affine_transform方法。由於這個question和awers指出，affine_transform方法可能有點不直觀（特別是關於偏移量計算），但是，用戶timday顯示如何應用旋轉和剪切圖像並將其放置在另一個數組中，而用戶地理數據給出更多背景信息。

我的問題是推廣那裏顯示的方法（1）爲圖像着色和（2）我計算自己的任意轉換。

這是我的代碼（這應該運行作爲您的計算機上）：

我試着包括與逆工作

import numpy as np 
from scipy import ndimage 
import matplotlib.pyplot as plt 


def calcAffineMatrix(sourcePoints, targetPoints): 
    # For three source- and three target points, find the affine transformation 
    # Function works correctly, not part of the question 
    A = [] 
    b = [] 
    for sp, trg in zip(sourcePoints, targetPoints): 
     A.append([sp[0], 0, sp[1], 0, 1, 0]) 
     A.append([0, sp[0], 0, sp[1], 0, 1]) 
     b.append(trg[0]) 
     b.append(trg[1]) 
    result, resids, rank, s = np.linalg.lstsq(np.array(A), np.array(b)) 

    a0, a1, a2, a3, a4, a5 = result 
    # Ignoring offset here, later use timday's suggested offset calculation 
    affineTrafo = np.array([[a0, a1, 0], [a2, a3, 0], [0, 0, 1]], 'd') 

    # Testing the correctness of transformation matrix 
    for i, _ in enumerate(sourcePoints): 
     src = sourcePoints[i] 
     src.append(1.) 
     trg = targetPoints[i] 
     trg.append(1.) 
     at = affineTrafo.copy() 
     at[2, 0:2] = [a4, a5] 
     assert(np.array_equal(np.round(np.array(src).dot(at)), np.array(trg))) 
    return affineTrafo 


# Prepare source image 
sourcePoints = [[162., 112.], [130., 112.], [162., 240.]] 
targetPoints = [[180., 102.], [101., 101.], [190., 200.]] 
image = np.empty((300, 300, 3), dtype='uint8') 
image[:] = 255 
# Mark border for better visibility 
image[0:2, :] = 0 
image[-3:-1, :] = 0 
image[:, 0:2] = 0 
image[:, -3:-1] = 0 
# Mark source points in red 
for sp in sourcePoints: 
    sp = [int(u) for u in sp] 
    image[sp[1] - 5:sp[1] + 5, sp[0] - 5:sp[0] + 5, :] = np.array([255, 0, 0]) 

# Show image 
plt.subplot(3, 1, 1) 
plt.imshow(image) 

# Prepare array in which the image is placed 
array = np.empty((400, 300, 3), dtype='uint8') 
array[:] = 255 
a2 = array.copy() 
# Mark target points in blue 
for tp in targetPoints: 
    tp = [int(u) for u in tp] 
    a2[tp[1] - 2:tp[1] + 2, tp[0] - 2:tp[0] + 2] = [0, 0, 255] 

# Show array 
plt.subplot(3, 1, 2) 
plt.imshow(a2) 

# Next 5 program lines are actually relevant for question: 

# Calculate affine matrix 
affineTrafo = calcAffineMatrix(sourcePoints, targetPoints) 

# This follows the c_in-c_out method proposed in linked stackoverflow issue 
# extended for color channel (no translation here) 
c_in = np.array([sourcePoints[0][0], sourcePoints[0][1], 0]) 
c_out = np.array([targetPoints[0][0], targetPoints[0][1], 0]) 
offset = (c_in - np.dot(c_out, affineTrafo)) 

# Affine transform! 
ndimage.interpolation.affine_transform(image, affineTrafo, order=2, offset=offset, 
             output=array, output_shape=array.shape, 
             cval=255) 
# Mark blue target points in array, expected to be above red source points 
for tp in targetPoints: 
    tp = [int(u) for u in tp] 
    array[tp[1] - 2:tp[1] + 2, tp[0] - 2:tp[0] + 2] = [0, 0, 255] 

plt.subplot(3, 1, 3) 
plt.imshow(array) 

plt.show() 

其他辦法，調換或兩者affineTrafo的：

affineTrafo = np.linalg.inv(affineTrafo) 
affineTrafo = affineTrafo.T 
affineTrafo = np.linalg.inv(affineTrafo.T) 
affineTrafo = np.linalg.inv(affineTrafo).T 

在他的回答中，地理數據顯示瞭如何計算affine_trafo需要進行縮放和旋轉的矩陣：

如果有人想先縮放S然後旋轉R，則認爲T=R*S因此T.inv=S.inv*R.inv（注意顛倒的順序）。

這點我嘗試使用矩陣分解（分解我的仿射變換成旋轉，剪切和另一個旋轉）來複制：

u, s, v = np.linalg.svd(affineTrafo[:2,:2]) 
uInv = np.linalg.inv(u) 
sInv = np.linalg.inv(np.diag((s))) 
vInv = np.linalg.inv(v) 
affineTrafo[:2, :2] = uInv.dot(sInv).dot(vInv) 

再次，沒有成功。

對於我的所有結果，它不是（唯一）一個抵消問題。從圖中清楚可見，源點和目標點的相對位置不一致。

我搜索了網絡和計算器，並沒有找到我的問題的答案。請幫幫我！ :)

Answer 1

我終於得到它的工作感謝AlexanderReynolds暗示使用另一個庫。這當然是一種解決方法;我無法使用scipy的affine_transform工作，所以我改用OpenCV cv2.warpAffine。在這種情況下是有幫助的任何人，這是我的代碼：

import numpy as np 
import matplotlib.pyplot as plt 
import cv2 

# Prepare source image 
sourcePoints = [[162., 112.], [130., 112.], [162., 240.]] 
targetPoints = [[180., 102.], [101., 101.], [190., 200.]] 
image = np.empty((300, 300, 3), dtype='uint8') 
image[:] = 255 
# Mark border for better visibility 
image[0:2, :] = 0 
image[-3:-1, :] = 0 
image[:, 0:2] = 0 
image[:, -3:-1] = 0 
# Mark source points in red 
for sp in sourcePoints: 
    sp = [int(u) for u in sp] 
    image[sp[1] - 5:sp[1] + 5, sp[0] - 5:sp[0] + 5, :] = np.array([255, 0, 0]) 

# Show image 
plt.subplot(3, 1, 1) 
plt.imshow(image) 

# Prepare array in which the image is placed 
array = np.empty((400, 300, 3), dtype='uint8') 
array[:] = 255 
a2 = array.copy() 
# Mark target points in blue 
for tp in targetPoints: 
    tp = [int(u) for u in tp] 
    a2[tp[1] - 2:tp[1] + 2, tp[0] - 2:tp[0] + 2] = [0, 0, 255] 

# Show array 
plt.subplot(3, 1, 2) 
plt.imshow(a2) 

# Calculate affine matrix and transform image 
M = cv2.getAffineTransform(np.float32(sourcePoints), np.float32(targetPoints)) 
array = cv2.warpAffine(image, M, array.shape[:2], borderValue=[255, 255, 255]) 

# Mark blue target points in array, expected to be above red source points 
for tp in targetPoints: 
    tp = [int(u) for u in tp] 
    array[tp[1] - 2:tp[1] + 2, tp[0] - 2:tp[0] + 2] = [0, 0, 255] 

plt.subplot(3, 1, 3) 
plt.imshow(array) 

plt.show() 

點評：

• 有趣的是如何幾乎立即改變工作的庫後。花了一天多的時間試圖讓它與scipy一起工作之後，這對我自己來說是更快地改變圖書館的一個教訓。
• 如果有人想找基於以上三點的仿射變換的（最小二乘近似），這是你如何得到與cv2.warpAffine作品矩陣：

代碼：

def calcAffineMatrix(sourcePoints, targetPoints): 
    # For three or more source and target points, find the affine transformation 
    A = [] 
    b = [] 
    for sp, trg in zip(sourcePoints, targetPoints): 
     A.append([sp[0], 0, sp[1], 0, 1, 0]) 
     A.append([0, sp[0], 0, sp[1], 0, 1]) 
     b.append(trg[0]) 
     b.append(trg[1]) 
    result, resids, rank, s = np.linalg.lstsq(np.array(A), np.array(b)) 

    a0, a1, a2, a3, a4, a5 = result 
    affineTrafo = np.float32([[a0, a2, a4], [a1, a3, a5]]) 
    return affineTrafo