该模型于 2025-08-14 贡献至 Hugging Face Transformers。

SAM2 视频

概述

SAM2（Segment Anything Model 2）由 Nikhila Ravi、Valentin Gabeur、Yuan-Ting Hu、Ronghang Hu、Chaitanya Ryali、Tengyu Ma、Haitham Khedr、Roman Rädle、Chloe Rolland、Laura Gustafson、Eric Mintun、Junting Pan、Kalyan Vasudev Alwala、Nicolas Carion、Chao-Yuan Wu、Ross Girshick、Piotr Dollár 和 Christoph Feichtenhofer 在 Segment Anything in Images and Videos 中提出。

给定输入图像或视频以及输入点或边界框，该模型可用于预测任意感兴趣对象的分割掩码。

example image

该论文的摘要如下：

我们提出了 Segment Anything Model 2（SAM 2），这是一个旨在解决图像和视频中的可提示视觉分割问题的基础模型。我们构建了一个数据引擎，通过用户交互来改进模型和数据，从而收集了迄今为止规模最大的视频分割数据集。我们的模型采用了一种简单的 Transformer 架构，并配备了用于实时视频处理的流式内存。在我们数据上训练的 SAM 2 在广泛的任务中均表现出强劲的性能。在视频分割方面，我们观察到其精度更高，且所需的交互次数比先前方法减少了 3 倍。在图像分割方面，我们的模型比 Segment Anything Model（SAM）更准确，速度快 6 倍。我们相信，我们的数据、模型和见解将成为视频分割及相关感知任务的重要里程碑。我们将发布我们模型的版本、数据集以及一个交互式演示。

提示：

批量与视频支持：SAM2 原生支持批量处理和无缝视频分割，而原始 SAM 专为静态图像和更简单的单张图像处理工作流而设计。
准确性与泛化能力：与原始 SAM 相比，SAM2 在分割质量、鲁棒性以及针对新领域的零样本泛化能力方面均有提升，尤其是在使用混合提示时。

该模型由 sangbumchoi 和 yonigozlan 贡献。原始代码可在此处找到。

使用示例

视频分割与跟踪

SAM2 的核心优势在于其跨视频帧跟踪对象的能力。以下是如何使用它进行视频分割：

基础视频跟踪

from transformers import Sam2VideoModel, Sam2VideoProcessor
import torch

model = Sam2VideoModel.from_pretrained("facebook/sam2.1-hiera-tiny", device_map="auto")
processor = Sam2VideoProcessor.from_pretrained("facebook/sam2.1-hiera-tiny")

# Load video frames (example assumes you have a list of PIL Images)
# video_frames = [Image.open(f"frame_{i:05d}.jpg") for i in range(num_frames)]

# For this example, we'll use the video loading utility
from transformers.video_utils import load_video
video_url = "https://huggingface.co/datasets/hf-internal-testing/sam2-fixtures/resolve/main/bedroom.mp4"
video_frames, _ = load_video(video_url)

# Initialize video inference session
inference_session = processor.init_video_session(
    video=video_frames,
    inference_device=device,
)

# Add click on first frame to select object
ann_frame_idx = 0
ann_obj_id = 1
points = [[[[210, 350]]]]
labels = [[[1]]]

processor.add_inputs_to_inference_session(
    inference_session=inference_session,
    frame_idx=ann_frame_idx,
    obj_ids=ann_obj_id,
    input_points=points,
    input_labels=labels,
)

# Segment the object on the first frame
outputs = model(
    inference_session=inference_session,
    frame_idx=ann_frame_idx,
)
video_res_masks = processor.post_process_masks(
    [outputs.pred_masks], original_sizes=[[inference_session.video_height, inference_session.video_width]], binarize=False
)[0]
print(f"Segmentation shape: {video_res_masks.shape}")
Segmentation shape: torch.Size([1, 1, 480, 854])

# Propagate through the entire video
video_segments = {}
for sam2_video_output in model.propagate_in_video_iterator(inference_session):
    video_res_masks = processor.post_process_masks(
        [sam2_video_output.pred_masks], original_sizes=[[inference_session.video_height, inference_session.video_width]], binarize=False
    )[0]
    video_segments[sam2_video_output.frame_idx] = video_res_masks

print(f"Tracked object through {len(video_segments)} frames")
Tracked object through 180 frames

多目标视频跟踪

在视频帧中同时跟踪多个对象：

# Reset for new tracking session
inference_session.reset_inference_session()

# Add multiple objects on the first frame
ann_frame_idx = 0
obj_ids = [2, 3]
input_points = [[[[200, 300]], [[400, 150]]]]  # Points for two objects (batched)
input_labels = [[[1], [1]]]

processor.add_inputs_to_inference_session(
    inference_session=inference_session,
    frame_idx=ann_frame_idx,
    obj_ids=obj_ids,
    input_points=input_points,
    input_labels=input_labels,
)

# Get masks for both objects on first frame
outputs = model(
    inference_session=inference_session,
    frame_idx=ann_frame_idx,
)

# Propagate both objects through video
video_segments = {}
for sam2_video_output in model.propagate_in_video_iterator(inference_session):
    video_res_masks = processor.post_process_masks(
        [sam2_video_output.pred_masks], original_sizes=[[inference_session.video_height, inference_session.video_width]], binarize=False
    )[0]
    video_segments[sam2_video_output.frame_idx] = {
        obj_id: video_res_masks[i]
        for i, obj_id in enumerate(inference_session.obj_ids)
    }

print(f"Tracked {len(inference_session.obj_ids)} objects through {len(video_segments)} frames")
Tracked 2 objects through 180 frames

优化视频分割

您可以在任意帧上添加额外的点击以优化跟踪：

# Add refinement click on a later frame
refine_frame_idx = 50
ann_obj_id = 2  # Refining first object
points = [[[[220, 280]]]]  # Additional point
labels = [[[1]]]  # Positive click

processor.add_inputs_to_inference_session(
    inference_session=inference_session,
    frame_idx=refine_frame_idx,
    obj_ids=ann_obj_id,
    input_points=points,
    input_labels=labels,
)

# Re-propagate with the additional information
video_segments = {}
for sam2_video_output in model.propagate_in_video_iterator(inference_session):
    video_res_masks = processor.post_process_masks(
        [sam2_video_output.pred_masks], original_sizes=[[inference_session.video_height, inference_session.video_width]], binarize=False
    )[0]
    video_segments[sam2_video_output.frame_idx] = video_res_masks

流式视频推理

对于实时应用，SAM2 支持在视频帧到达时进行处理：

# Initialize session for streaming
inference_session = processor.init_video_session(
    inference_device=device,
)

# Process frames one by one
for frame_idx, frame in enumerate(video_frames[:10]):  # Process first 10 frames
    inputs = processor(images=frame, device=device, return_tensors="pt").to(model.device)
...
    if frame_idx == 0:
        # Add point input on first frame
        processor.add_inputs_to_inference_session(
            inference_session=inference_session,
            frame_idx=0,
            obj_ids=1,
            input_points=[[[[210, 350], [250, 220]]]],
            input_labels=[[[1, 1]]],
            original_size=inputs.original_sizes[0], # need to be provided when using streaming video inference
        )
...
    # Process current frame
    sam2_video_output = model(inference_session=inference_session, frame=inputs.pixel_values[0])
...
    video_res_masks = processor.post_process_masks(
        [sam2_video_output.pred_masks], original_sizes=inputs.original_sizes, binarize=False
    )[0]
    print(f"Frame {frame_idx}: mask shape {video_res_masks.shape}")

多对象视频批量处理

通过一次性添加所有对象，在视频中同时跟踪多个对象：

# Initialize video session
inference_session = processor.init_video_session(
    video=video_frames,
    inference_device=device,
)

# Add multiple objects on the first frame using batch processing
ann_frame_idx = 0
obj_ids = [2, 3]  # Track two different objects
input_points = [
    [[[200, 300], [230, 250], [275, 175]], [[400, 150]]]
]  # Object 2: 3 points (2 positive, 1 negative); Object 3: 1 point
input_labels = [
    [[1, 1, 0], [1]]
]  # Object 2: positive, positive, negative; Object 3: positive

processor.add_inputs_to_inference_session(
    inference_session=inference_session,
    frame_idx=ann_frame_idx,
    obj_ids=obj_ids,
    input_points=input_points,
    input_labels=input_labels,
)

# Get masks for all objects on the first frame
outputs = model(
    inference_session=inference_session,
    frame_idx=ann_frame_idx,
)
video_res_masks = processor.post_process_masks(
    [outputs.pred_masks], original_sizes=[[inference_session.video_height, inference_session.video_width]], binarize=False
)[0]
print(f"Generated masks for {video_res_masks.shape[0]} objects")
Generated masks for 2 objects

# Propagate all objects through the video
video_segments = {}
for sam2_video_output in model.propagate_in_video_iterator(inference_session):
    video_res_masks = processor.post_process_masks(
        [sam2_video_output.pred_masks], original_sizes=[[inference_session.video_height, inference_session.video_width]], binarize=False
    )[0]
    video_segments[sam2_video_output.frame_idx] = {
        obj_id: video_res_masks[i]
        for i, obj_id in enumerate(inference_session.obj_ids)
    }

print(f"Tracked {len(inference_session.obj_ids)} objects through {len(video_segments)} frames")
Tracked 2 objects through 180 frames

资源

一份来自 Hugging Face 官方和社区（以 🌎 标识）的资源列表，帮助你快速上手 SAM。

由 Sangbum Choi 贡献的用于该模型的 Demo notebook 🌎。

Transformers

SAM2 视频

概述

使用示例

视频分割与跟踪

基础视频跟踪

多目标视频跟踪

优化视频分割

流式视频推理

多对象视频批量处理

资源

Sam2VideoConfig

class transformers.Sam2VideoConfig

Sam2VideoMaskDecoderConfig

类 transformers.Sam2VideoMaskDecoderConfig

Sam2VideoPromptEncoderConfig

类 transformers.Sam2VideoPromptEncoderConfig

Sam2VideoProcessor

class transformers.Sam2VideoProcessor

__call__

post_process_masks

init_video_session

add_inputs_to_inference_session

Sam2VideoVideoProcessor

class transformers.Sam2VideoVideoProcessor

post_process_masks

Sam2VideoInferenceSession

class transformers.Sam2VideoInferenceSession

add_mask_inputs

add_new_frame

add_point_inputs

get_frame

get_obj_num

get_output

obj_id_to_idx

obj_idx_to_id

remove_mask_inputs

remove_point_inputs

reset_inference_session

reset_tracking_data

store_output

Sam2VideoModel

class transformers.Sam2VideoModel

forward

propagate_in_video_iterator

get_image_features

call