Reproducing the issue on a different system

Oct 13, 2025 by thesweetdesigns.com 59 views

🐛 Describe the bug

When using the FSDP2 API to shard a model, extract its state dictionary, and then completely remove the model from memory, a memory leak occurs. This issue is demonstrated using a test case from the PyTorch repository.

Problem Statement

The problem arises when attempting to use the get_model_state_dict function to extract the state dictionary of a model that has been fully sharded using the FSDP API. The state dictionary is not properly released, resulting in a memory leak.

Versions

PyTorch version: 2.6.0+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A
OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31
Python version: 3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-163-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
- GPU 0: NVIDIA A100-SXM4-80GB
- GPU 1: NVIDIA A100-SXM4-80GB
- GPU 2: NVIDIA A100-SXM4-80GB
- GPU 3: NVIDIA A100-SXM4-80GB
- GPU 4: NVIDIA A100-SXM4-80GB
- GPU 5: NVIDIA A100-SXM4-80GB
- GPU 6: NVIDIA A100-SXM4-80GB
- GPU 7: NVIDIA A100-SXM4-80GB

Reproducing the issue

To reproduce the issue, you can use the following code snippet:

import gc

import torch
from torch.distributed.fsdp import fully_shard
from torch.testing._internal.common_fsdp import FSDPTest
from torch.testing._internal.common_utils import run_tests
from torch.testing._internal.distributed._tensor.common_dtensor import (
    ModelArgs,
    Transformer,
    TransformerBlock,
)
import os
import torch
import gc

from torch.distributed import init_process_group
from datetime import timedelta
from torch.distributed.checkpoint.state_dict import get_model_state_dict, StateDictOptions


class TestFullyShardMemory(FSDPTest):
    @property
    def world_size(self) -> int:
        return min(2, torch.cuda.device_count())
    def _get_peak_active_memory_mb(self) -> int:
        mem_stats = torch.cuda.memory_stats()
        return round(mem_stats["active_bytes.all.peak"] / 1e6)

    def _get_curr_active_memory_mb(self) -> int:
        mem_stats = torch.cuda.memory_stats()
        return round(mem_stats["active_bytes.all.current"] / 1e6)

    def test_fully_shard_del_memory(self):
        base_mem_mb = self._get_peak_active_memory_mb()
        vocab_size = 32
        model_args = ModelArgs(
            vocab_size=vocab_size, n_layers=3, dim=768, n_heads=12, weight_tying=False
        )
        model = Transformer(model_args)
        # Initializing the model on CPU should not change the GPU memory usage
        post_model_init_mem_mb = self._get_peak_active_memory_mb()
        self.assertEqual(base_mem_mb, post_model_init_mem_mb)

        for module in model.modules():
            if isinstance(module, TransformerBlock):
                fully_shard(module)
        fully_shard(model)
        unsharded_numel = sum(p.numel() for p in model.parameters())
        sharded_numel = unsharded_numel // self.world_size
        buffer_mb = 4
        mem_mb = self._get_curr_active_memory_mb()
        expected_mb = sharded_numel * 4 / 1e6 + buffer_mb
        self.assertLessEqual(mem_mb - base_mem_mb, expected_mb)
        
        ### DIFF STARTS HERE ###
        
        sdo = StateDictOptions(full_state_dict=True, cpu_offload=True, broadcast_from_rank0=True)         
        state_dict =  get_model_state_dict(model, options=sdo)
        del state_dict
        
        ### DIFF ENDS HERE ### 
        # Deleting the model should free all of the FSDP-managed GPU memory
        del model
        # Manually call garbage collection since there are ref cycles in FSDP
        gc.collect()
        torch.cuda.empty_cache()
        mem_mb = self._get_curr_active_memory_mb()
        print(f"Mem MB: {mem_mb}")
        print(f"Base Mem MB: {base_mem_mb}")
        self.assertEqual(mem_mb, base_mem_mb)

if __name__ == "__main__":
    init_process_group(backend="nccl", timeout=timedelta(hours=24))

    dst_rank = int(os.environ['RANK'])
    dst_local_rank = int(os.environ['LOCAL_RANK'])
    dst_world_size = int(os.environ['WORLD_SIZE'])
    device = f'cuda:{dst_local_rank}'
    run_tests()

Expected behavior

The expected behavior is that the memory usage should decrease after deleting the model and its state dictionary.

Actual behavior

The actual behavior is that the memory usage remains the same, indicating a memory leak.

Conclusion

The issue arises from the fact that the get_model_state_dict function does not properly release the state dictionary, resulting in a memory leak. This issue needs to be addressed to ensure that the memory usage is properly released after deleting the model and its state dictionary.

Workaround

A workaround for this issue is to manually call the del statement on the state dictionary to release its memory.

Future work

In the future, it would be beneficial to investigate the root cause of this issue and provide a more robust solution to ensure that the memory usage is properly released after deleting the model and its state dictionary.

Related issues

This issue is related to the following issues:

Additional information

The following information may be helpful in reproducing the issue:

PyTorch version: 2.6.0+cu124
CUDA version: 12.4
GPU models and configuration:
- GPU 0: NVIDIA A100-SXM4-80GB
- GPU 1: NVIDIA A100-SXM4-80GB
- GPU 2: NVIDIA A100-SXM4-80GB
- GPU 3: NVIDIA A100-SXM4-80GB
- GPU 4: NVIDIA A100-SXM4-80GB
- GPU 5: NVIDIA A100-SXM4-80GB
- GPU 6: NVIDIA A100-SXM4-80GB
- GPU 7: NVIDIA A100-SXM4-80GB

Reproducing the issue on a different system

To reproduce the issue on a different system, you can follow these steps:

Install PyTorch version 2.6.0+cu124.
Install CUDA version 12.4.
Install the necessary GPU models and configuration.
Run the code snippet provided above.

Q: What is the issue with using get_model_state_dict with FSDP-sharded models?

A: The issue arises from the fact that the get_model_state_dict function does not properly release the state dictionary, resulting in a memory leak.

Q: What is the expected behavior when deleting the model and its state dictionary?

A: The expected behavior is that the memory usage should decrease after deleting the model and its state dictionary.

Q: What is the actual behavior when deleting the model and its state dictionary?

A: The actual behavior is that the memory usage remains the same, indicating a memory leak.

Q: How can I reproduce the issue?

A: You can reproduce the issue by running the code snippet provided above.

Q: What are the system requirements to reproduce the issue?

A: The system requirements are:

PyTorch version: 2.6.0+cu124
CUDA version: 12.4
GPU models and configuration:
- GPU 0: NVIDIA A100-SXM4-80GB
- GPU 1: NVIDIA A100-SXM4-80GB
- GPU 2: NVIDIA A100-SXM4-80GB
- GPU 3: NVIDIA A100-SXM4-80GB
- GPU 4: NVIDIA A100-SXM4-80GB
- GPU 5: NVIDIA A100-SXM4-80GB
- GPU 6: NVIDIA A100-SXM4-80GB
- GPU 7: NVIDIA A100-SXM4-80GB

Q: How can I fix the issue?

A: A workaround for this issue is to manually call the del statement on the state dictionary to release its memory.

Q: What are the related issues?

A: The related issues are:

Q: What is the future work?

A: In the future, it would be beneficial to investigate the root cause of this issue and provide a more robust solution to ensure that the memory usage is properly released after deleting the model and its state dictionary.

Q: What is the additional information?

A: The following information may be helpful in reproducing the issue:

PyTorch version: 2.6.0+cu124
CUDA version: 12.4
GPU models and configuration:
- GPU 0: NVIDIA A100-SXM4-80GB
- GPU 1: NVIDIA A100-SXM4-80GB
- GPU 2: NVIDIA A100-SXM4-80GB
- GPU 3: NVIDIA A100-SXM4-80GB
- GPU 4: NVIDIA A100-SXM4-80GB
- GPU 5: NVIDIA A100-SXM4-80GB
- GPU 6: NVIDIA A100-SXM4-80GB
- GPU 7: NVIDIA A100-SXM4-80GB

Q: How can I reproduce the issue on a different system?

A: To reproduce the issue on a different system, you can follow these steps:

Install PyTorch version 2.6.0+cu124.
Install CUDA version 12.4.
Install the necessary GPU models and configuration.
Run the code snippet provided above.

Note: The issue may not be reproducible on all systems, and the symptoms may vary depending on the specific system configuration.