What does .contiguous() do in PyTorch?

In PyTorch, the tensor’s memory layout can impact the computational efficiency. When elements are stored continuously in a block of memory, we call it a contiguous tensor. The continuous arrangement makes the iterating over the tensor element efficient.

What is contiguous()?

The use of .contiguous() the method in PyTorch is to ensure that memory is laid out in a contiguous manner. It returns a contiguous copy of the tensor, rearranging the data if they are already in a contiguous manner.

When do we use contiguous()?

When we perform certain operations such as transpose() or view() or slicing the tensor may become non-contiguous and change the tensor’s shape, or the order of elements without modifying the underlying data in memory. This results in the loss of contiguous layout.

For example, consider a tensor A and its transpose A.T. While A might be contiguous, A.T is typically non-contiguous because the memory layout does not change during transpose – only the way it is indexed changes.

import torch

# Creating a tensor
A = torch.rand(3, 4)

# Transposing the tensor
B = A.transpose(0, 1)

# Checking if B is contiguous
print("Is B contiguous? ", B.is_contiguous())

# Making B contiguous
C = B.contiguous()

# Checking if C is now contiguous
print("Is C contiguous? ", C.is_contiguous())

In the above example, After applying the transpose(), B became non-contiguous. Using .contiguous(), we create a contiguous tensor C from B.

Frequently asked questions:

1. Does .contiguous() always create a copy of the tensor?
   No, .contiguous() only creates a copy if the tensor is non-contiguous. Otherwise, it returns the original tensor.
2. How does non-contiguity affect tensor operations?
   Non-contiguous tensors can lead to inefficient memory access patterns, slowing down operations and computations.
3. Can .contiguous() affect the values in a tensor?
   No, .contiguous() does not change the tensor’s values; it only affects the memory layout.
4. Is it necessary to use .contiguous() after every operation like transpose()?
   It depends on your subsequent operations. If an operation requires contiguous tensors, then .contiguous() should be used.
5. How do I check if a tensor is contiguous?
   Use the is_contiguous() method on a tensor to check its contiguity.