README.md

metadata

title: SAM3 Promptable Concept Segmentation
emoji: 🎯
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
license: apache-2.0
short_description: SAM3 inference with text prompts and SAM2 API compatibility

SAM3 Promptable Concept Segmentation

This Space provides both a web interface and REST API for SAM3 (Segment Anything Model 3) inference, featuring:

🚀 Key Features

• 🆕 Text Prompts: Segment objects using natural language descriptions (e.g., "kitten", "car", "person wearing red shirt")
• 🔄 SAM2 Compatible: Drop-in replacement for existing SAM2 inference endpoints
• 📊 High Quality: Uses official SAM3 post-processing for single high-confidence masks
• 🔌 Dual APIs: Simple Gradio API + SAM2-compatible inference endpoint format
• ⚡ Fast: Optimized for production use with proper confidence thresholding

📖 Usage

Web Interface

Simply upload an image, enter a text description of what you want to segment, and adjust the confidence threshold.

API Usage

1. Simple Text API (Gradio format)

import requests
import base64

# Encode your image to base64
with open("image.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

# Make API request
response = requests.post(
    "https://your-username-sam3-api.hf.space/api/predict",
    json={
        "data": [image_b64, "kitten", 0.5]
    }
)

result = response.json()

2. SAM2/SAM3 Compatible API (Inference Endpoint format)

import requests
import base64

# Encode your image to base64
with open("image.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

# SAM3 Text Prompts (NEW)
response = requests.post(
    "https://your-username-sam3-api.hf.space/api/sam2_compatible",
    json={
        "data": [{
            "inputs": {
                "image": image_b64,
                "text_prompts": ["kitten", "toy"],
                "confidence_threshold": 0.5
            }
        }]
    }
)

# SAM2 Compatible (Points/Boxes)
response = requests.post(
    "https://your-username-sam3-api.hf.space/api/sam2_compatible",
    json={
        "data": [{
            "inputs": {
                "image": image_b64,
                "boxes": [[100, 100, 200, 200]],
                "confidence_threshold": 0.5
            }
        }]
    }
)

result = response.json()

🔧 API Parameters

SAM2-Compatible API Input

{
  "inputs": {
    "image": "base64_encoded_image_string",

    // SAM3 NEW: Text-based prompts
    "text_prompts": ["person", "car"],  // List of text descriptions

    // SAM2 COMPATIBLE: Point-based prompts
    "points": [[[x1, y1]], [[x2, y2]]],  // Points for each object
    "point_labels": [[1], [1]],  // Labels for each point (1=foreground, 0=background)

    // SAM2 COMPATIBLE: Bounding box prompts
    "boxes": [[x1, y1, x2, y2], [x1, y1, x2, y2]],  // Bounding boxes
    "box_labels": [1, 0],  // Labels for each box (1=positive, 0=negative/exclude)

    "multimask_output": false,  // Optional, defaults to False
    "confidence_threshold": 0.5  // Optional, minimum confidence for returned masks
  }
}

API Response

{
  "masks": ["base64_encoded_mask_1", "base64_encoded_mask_2"],
  "scores": [0.95, 0.87],
  "num_objects": 2,
  "sam_version": "3.0",
  "success": true
}

🆚 SAM3 vs SAM2

Feature	SAM2	SAM3
Text Prompts	❌	✅ Natural language descriptions
Point Prompts	✅	✅ (compatible)
Box Prompts	✅	✅ (compatible)
Quality	High	Higher (concept-aware)
API Format	HF Inference Endpoints	✅ Compatible + Extensions

🔬 Technical Details

• Model: facebook/sam3 from HuggingFace Transformers
• Post-processing: Official post_process_instance_segmentation() API
• Framework: Gradio 5.49.1 with automatic API generation
• Dependencies: Latest transformers with SAM3 support
• Deployment: HuggingFace Spaces (avoids Inference Toolkit compatibility issues)

📚 References

• SAM3 Model Card
• SAM3 Paper
• Transformers SAM3 Documentation