So far, the kernels we’ve examined match a single template against a single location in the search image. In practice, an application may require searching an image against many templates at once – a bank of candidate features, or one template rendered at many rotations and scales. For such applications, it turns out Tensor Cores’ native support for INT8 matrix multiply proffers ideal hardware support.
Computing the numerator for a whole bank at once can be expressed as a matrix multiply. The im2col transform lowers each output window to a row of \(K = T \times T\) pixels, giving an \(M \times K\) matrix \(A\) with one row per output pixel; the \(N\) templates form the columns of a \(K \times N\) matrix \(B\); and \(\text{SumIT}[p][t] = \sum_{\text{window}} I \cdot T_t = (AB)[p][t]\) is an \(M{\times}K\) by \(K{\times}N\) GEMM. A single template is the degenerate case \(N = 1\) – a matrix–vector product that leaves the Tensor Cores nearly idle – while a bank of \(N \ge 16\) fills the tile, and the whole search runs as one INT8 Tensor-Core GEMM through cuBLAS.
INT8 keeps the products exact – a pixel times a pixel is at most
\(255^2\), and \(K\) of them sum to well under \(2^{31}\) – but the format is signed and the
pixels are not. CUDA_R_8I reads its operands as signed
\([-128, 127]\), while grayscale pixels
are unsigned \([0, 255]\); passed to
the GEMM as they are, every pixel above 127 is read as negative and the
products are wrong. The remedy is to run the GEMM on the centered
operands \((I-128)\) and \((T-128)\) and correct for the offset
afterward. Expanding the centered product gives
\[\text{SumIT} = \text{GEMM}(I-128,\ T-128) + 128\sum I + 128\sum T - 128^2 K,\]
where the three correction terms are sums the coefficient already
computes, so the correction does not add any arithmetic. Reconstructed
this way, SumIT matches a direct spatial baseline to the
bit.
Forming the coefficient from the sums has a second pitfall, specific
to consumer GPUs. Written the natural way – in double, with
a sqrt – it cost more than the GEMM itself on a GeForce,
because consumer parts run double precision at 1/32 to 1/64 of their
single-precision rate. The computation does not need
double: the numerator and the two variances are differences
of products of the sums, all exact in int64, where integer
arithmetic runs at full rate and the exact subtraction also avoids the
cancellation a float combine suffers on low-variance
windows. Only the final ratio needs floating point, formed with
rsqrtf of each variance. The int64 combine
measured about ten times faster than the double one and
returned the Tensor-Core GEMM to being the pipeline’s limiter. As a rule
on consumer GPUs: keep double out of device code,
accumulate exactly in int64, and drop to float
only for the final reciprocal square root.
On a GeForce RTX 3060, matching 32 templates of 16×16 against the
coins image, the INT8 Tensor-Core GEMM computes the numerator at about
5460 GOP/s – roughly 7.3 times the rate of a hand-written kernel that
reuses each window across the bank without Tensor Cores (about 750
GOP/s). The sample is corrBankTC.cu.