Guide to Benchmarking

The performance of Python applications that use TACO can be measured using Python's built-in time.perf_counter function with minimal changes to the applications. As an example, we can benchmark the performance of the scientific computing application shown here as follows:

import pytaco as pt
from pytaco import compressed, dense
import numpy as np
import time

csr = pt.format([dense, compressed])
dv  = pt.format([dense])

A = pt.read("pwtk.mtx", csr)
x = pt.from_array(np.random.uniform(size=A.shape[1]))
z = pt.from_array(np.random.uniform(size=A.shape[0]))
y = pt.tensor([A.shape[0]], dv)

i, j = pt.get_index_vars(2)
y[i] = A[i, j] * x[j] + z[i]

# Tell TACO to generate code to perform the SpMV computation
y.compile()

# Benchmark the actual SpMV computation
start = time.perf_counter()
y.compute()
end = time.perf_counter()

print("Execution time: {0} seconds".format(end - start))

In order to accurately measure TACO's computational performance, only the time it takes to actually perform a computation should be measured. The time it takes to generate code under the hood for performing that computation should not be measured, since this overhead can be quite variable but can often be amortized in practice. By default though, TACO will only generate and compile code it needs for performing a computation immediately before it has to actually perform the computation. As the example above demonstrates, by manually calling the result tensor's compile method, we can tell TACO to generate code needed for performing the computation before benchmarking starts, letting us measure only the performance of the computation itself.

The time it takes to construct the initial operand tensors should also not be measured, since again this overhead can often be amortized in practice. By default, pytaco.read and functions for converting NumPy arrays and SciPy matrices to TACO tensors return fully constructed tensors. If you add nonzero elements to an operand tensor by invoking its insert method though, then pack must also be explicitly invoked before any benchmarking is done:

import pytaco as pt
from pytaco import compressed, dense
import numpy as np
import random
import time

csr = pt.format([dense, compressed])
dv  = pt.format([dense])

A = pt.read("pwtk.mtx", csr)
x = pt.tensor([A.shape[1]], dv)
z = pt.tensor([A.shape[0]], dv)
y = pt.tensor([A.shape[0]], dv)

# Insert random values into x and z and pack them into dense arrays
for k in range(A.shape[1]):
  x.insert([k], random.random())
x.pack()
for k in range(A.shape[0]):
  z.insert([k], random.random())
z.pack()

i, j = pt.get_index_vars(2)
y[i] = A[i, j] * x[j] + z[i]

y.compile()

start = time.perf_counter()
y.compute()
end = time.perf_counter()

print("Execution time: {0} seconds".format(end - start))

TACO avoids regenerating code for performing the same computation though as long as the computation is redefined with the same index variables and with the same operand and result tensors. Thus, if your application executes the same computation many times in a loop and if the computation is executed on sufficiently large data sets, TACO will naturally amortize the overhead associated with generating code for performing the computation. In such scenarios, it is acceptable to include the initial code generation overhead in the performance measurement:

import pytaco as pt
from pytaco import compressed, dense
import numpy as np
import time

csr = pt.format([dense, compressed])
dv  = pt.format([dense])

A = pt.read("pwtk.mtx", csr)
x = pt.tensor([A.shape[1]], dv)
z = pt.tensor([A.shape[0]], dv)
y = pt.tensor([A.shape[0]], dv)

for k in range(A.shape[1]):
  x.insert([k], random.random())
x.pack()
for k in range(A.shape[0]):
  z.insert([k], random.random())
z.pack()

i, j = pt.get_index_vars(2)

# Benchmark the iterative SpMV computation, including overhead for 
# generating code in the first iteration to perform the computation
start = time.perf_counter()
for k in range(1000):
  y[i] = A[i, j] * x[j] + z[i]
  y.evaluate()
  x[i] = y[i]
  x.evaluate()
end = time.perf_counter()

print("Execution time: {0} seconds".format(end - start))