PyTorch Bug: Corrupted Tensors On Failed Storage Resize

If you're a deep learning enthusiast or a seasoned machine learning engineer working with PyTorch, you've likely encountered its incredible flexibility when it comes to handling tensors. Tensors are the fundamental building blocks of deep learning models, and PyTorch offers a robust set of tools to manipulate them. However, even the most sophisticated libraries can have their quirks, and a recent discovery highlights a particularly nasty bug in PyTorch related to tensor storage and resizing. This article delves into the specifics of this PyTorch tensor metadata corruption issue, explaining how it occurs, its potential consequences, and what it means for your workflows.

Understanding Tensor Storage and Resizing in PyTorch

Before we dive into the bug, let's quickly recap what tensor storage and resizing mean in PyTorch. A tensor in PyTorch is essentially a multidimensional array. Under the hood, it has two main components: the data (storage) and the metadata (shape, stride, and offset). The storage is the actual block of memory holding the tensor's elements, while the metadata describes how to interpret that memory as a multidimensional array. PyTorch's tensor operations often involve resizing or reshaping tensors. The resize_() method, for instance, attempts to change the number of elements in a tensor, which can involve reallocating memory if the underlying storage is dynamic.

What happens when a tensor's storage isn't meant to be resized? PyTorch has mechanisms to handle this. For example, if you create a tensor directly from a NumPy array using torch.from_numpy(), the resulting PyTorch tensor might share its storage with the NumPy array. NumPy arrays often have fixed-size storage, meaning they cannot be resized after creation. In such cases, attempting to resize the PyTorch tensor using resize_() should ideally fail gracefully, informing you that the storage is not resizable and leaving the tensor's metadata intact. This is where the bug we're discussing comes into play.

The Core of the Problem: Exception-Unsafe Resize

The issue arises when resize_() is called on a tensor whose storage is not resizable. PyTorch does detect this situation and correctly raises a RuntimeError with the informative message: "Trying to resize storage that is not resizable." This is a good thing – it prevents unexpected behavior. However, the problem lies in the fact that the operation is exception-unsafe. According to the expected behavior, when an error occurs during an operation, the system should ideally roll back to its previous state, ensuring that no partial changes are made. This is known as a strong exception guarantee. In this PyTorch bug, that guarantee is broken.

What actually happens is that PyTorch updates the tensor's shape and stride metadata to reflect the new target size before it checks if the storage can actually accommodate this change. When the check fails (because the storage is immutable), it raises the RuntimeError. But by then, the tensor's metadata has already been altered. This leads to a critical inconsistency: the tensor's shape metadata might indicate a large, new size (e.g., torch.Size([5, 5, 5])), while its underlying storage remains unchanged and, crucially, empty (0 bytes). This creates what the reporter aptly calls a "Zombie" tensor – it looks like it has data, but it actually doesn't, and accessing it leads to predictable chaos.

The Devastating Consequences: Crashes and Corrupted Data

This