Bot-Derpy/racquet-sports-analyzer

🏸 Racquet Sports Video Analyzer

PBVision / SwingVision style AI analysis for any racquet sport video.

Automatically detects rallies, tracks players, classifies shots, and generates complete player performance profiles — from any video angle (portrait or landscape).

Supported Sports

Sport	Ball Tracking	Shot Types	Court Detection
🏸 Badminton	✅	15 types (smash, drop, clear, ...)	✅
🎾 Tennis	✅	12 types (forehand, backhand, serve, ...)	✅
🥒 Pickleball	✅	11 types (dink, third-shot-drop, erne, ...)	✅
🏓 Padel	✅	11 types (bandeja, vibora, bajada, ...)	✅

What It Does

INPUT: Any racquet sport video (phone recording, broadcast, GoPro)
                    ↓
┌──────────────────────────────────────────┐
│  Court Detection (lines, net, homography)│
│  Player Detection + Tracking (YOLO11)    │
│  Ball/Shuttle Tracking (CV + Kalman)     │
│  Hit Detection (ball reversal + wrist)   │
│  Rally Segmentation (temporal grouping)  │
│  Shot Classification (VideoMAE)          │
│  Player Analytics (movement, shots, etc) │
└──────────────────────────────────────────┘
                    ↓
OUTPUT:
  📼 Annotated video with overlays
  📊 JSON report with full match analysis
  🗺️ Player movement heatmaps
  👤 Player profiles (playstyle, strengths/weaknesses)

Quick Start

1. Install

pip install -r requirements.txt

2. Run on your video

# Auto-detect sport, GPU mode
python run_analysis.py --video match.mp4

# Specify sport + output dir
python run_analysis.py --video match.mp4 --sport badminton --output ./my_analysis

# Portrait video from phone (pickleball at the park)
python run_analysis.py --video phone_recording.mp4 --sport pickleball

# CPU mode (slower but works everywhere)
python run_analysis.py --video match.mp4 --device cpu

# Fast mode: skip frames + no shot classification
python run_analysis.py --video match.mp4 --frame-skip 2 --no-shot-classification

3. Python API

from racquet_sports_analyzer import RacquetSportsAnalyzer, PipelineConfig, Sport

config = PipelineConfig().for_sport(Sport.BADMINTON)
analyzer = RacquetSportsAnalyzer(config)
results = analyzer.analyze("match.mp4", output_dir="./output")

# Access results
print(f"Rallies: {results['match_summary']['total_rallies']}")
for pid, profile in results['player_profiles'].items():
    print(f"Player {pid}: {profile['playstyle']} — {profile['rally']['win_rate']:.0%} win rate")

Output: Player Profile Example

==================================================
  PLAYER 1 PROFILE
==================================================
  Side: Near/Top
  Playstyle: AGGRESSIVE
  Time tracked: 142.3s (4269 frames)

  📍 MOVEMENT
     Distance: 48523px (89.2m)
     Avg speed: 341 px/s (6.3 m/s)
     Max speed: 1205 px/s (22.1 m/s)
     Coverage: 67.2%
     Dominant side: balanced
     Idle time: 18.4%

  🏸 SHOTS
     Total: 47
     Favorite: smash
     smash: 31.9%
     clear: 21.3%
     drop: 17.0%

  🎯 RALLY PERFORMANCE
     Played: 23
     Won: 15 | Lost: 8
     Win rate: 65%
     Avg length when winning: 4.2 shots
     Avg length when losing: 7.1 shots

  💪 BIOMECHANICS
     Max wrist speed: 28.4 px/frame
     Avg elbow angle: 142.3°

  📊 PLAYSTYLE BREAKDOWN
     aggressive         ████████████████░░░░ 62.5%
     counter-puncher    ████░░░░░░░░░░░░░░░░ 15.0%
     all-round          ███░░░░░░░░░░░░░░░░░ 12.5%
     defensive          ██░░░░░░░░░░░░░░░░░░ 10.0%

  ✅ STRENGTHS
     • Excellent court coverage
     • Explosive court speed
     • Strong smash
     • Diverse shot selection
     • High win rate (65%)
     • 3 first-shot winners
  ⚠️  WEAKNESSES
     • Struggles in long rallies
==================================================

Architecture Deep Dive

Models Used

Component	Model	Source	Purpose
Player Detection + Tracking	YOLO11n-pose + ByteTrack	ultralytics	Real-time person detection with pose keypoints
Ball Tracking	Classical CV + Kalman Filter	Built-in	Motion + color + shape filtering for ball detection
Shot Classification	VideoMAE-base	MCG-NJU/videomae-base	16-frame clip → shot type label
Pose (high-accuracy)	ViTPose+	usyd-community/vitpose-base-simple	COCO-17 keypoint estimation (optional upgrade)

Pipeline Flow (per frame)

# 1. Court Detection (every 30 frames)
court = court_detector.detect(frame)          # lines, net, homography

# 2. Player Detection + Tracking  
players = player_detector.detect_and_track(frame)  # YOLO11 + ByteTrack
# → track_id, bbox, center, feet_position, keypoints(17×2)

# 3. Ball Tracking
ball = ball_tracker.track(frame)              # BG subtraction + color + Kalman
# → position, velocity, radius, confidence

# 4. Hit Detection (dual signal fusion)
ball_hit = ball_tracker.detect_hit_event()    # ball direction reversal
wrist_swing = player_detector.detect_swing(track_id)  # wrist velocity spike
hit_confirmed = ball_hit OR wrist_swing       # fused signal

# 5. Rally Detection
rally = rally_detector.process_frame(         # temporal grouping
    ball_hit, ball_position, players, wrist_swings
)
# → rally start/end, shot count, serve team, winner

# 6. Shot Classification (on hit events)
if hit_confirmed:
    clip = shot_classifier.extract_hit_clip(frames, hit_frame)  # 3 pre + 12 post
    shot_type = shot_classifier.classify_from_frames(clip)
    # → "smash", "drop", "clear", etc.

# 7. Analytics (post-processing)
for player in tracked_players:
    profile = analytics.analyze_player(track_history, rallies)
    # → movement, shots, rally stats, biomechanics, playstyle

Research Foundation

This system is built on these SOTA papers:

Paper	What We Use	Citation
TOTNet	Ball tracking architecture (3D conv U-Net + optical flow)	arxiv:2508.09650
WASB	Multi-sport ball detection baseline (HRNet)	arxiv:2311.05237
BST	Shot classification strategy (hit-centered clips + skeleton + trajectory)	arxiv:2502.21085
BFMD	Full pipeline design (YOLOX + OC-SORT + TrackNetV2 + VideoMAE)	arxiv:2603.25533
ByteTrack	Player tracking (associates all detection boxes incl. low-score)	arxiv:2110.06864
GTA	Tracklet post-processing for long videos	arxiv:2411.08216
VideoMAE	Action recognition backbone for shot classification	arxiv:2203.12602
ViTPose+	High-accuracy pose estimation (81.1 AP on COCO)	arxiv:2212.04246

Improving Accuracy

Level 1: Out-of-the-box (this repo)

• Classical ball tracking + YOLO detection + zero-shot VideoMAE
• Good for: quick analysis, recreational play, phone recordings

Level 2: Fine-tune shot classifier

from racquet_sports_analyzer.shot_classifier import ShotClassifierTrainer

trainer = ShotClassifierTrainer(sport=Sport.BADMINTON)
trainer.train(
    train_clips=my_labeled_clips,    # list of (16_frames, label_idx) 
    val_clips=my_val_clips,
    output_dir="./shot_model_badminton",
    epochs=30,
    batch_size=16,
    push_to_hub=True,
    hub_model_id="your-username/shot-classifier-badminton",
)

• Use ShuttleSet (36K labeled strokes) for badminton
• Use TenniSet (3.5K strokes) for tennis
• For pickleball/padel: collect and label 30-50 matches

Level 3: Train deep ball tracker (TOTNet/WASB)

• Replace classical CV ball tracker with trained neural network
• Download TrackNetV2 dataset (26 badminton matches, NCTU)
• Train TOTNet: github.com/AugustRushG/TOTNet
• Or WASB: github.com/nttcom/WASB-SBDT
• Set config.ball_tracking.method = "tracknet" and provide model path

Level 4: ViTPose+ for biomechanics

config.pose.method = "vitpose"
config.pose.vitpose_checkpoint = "usyd-community/vitpose-plus-base"
# → Higher accuracy wrist/elbow tracking → better shot detection + biomechanics

Hardware Requirements

Mode	GPU	Speed	Quality
CPU (laptop)	None	~2-5 fps	Good (no shot classification)
T4 (Colab)	16GB	~15-25 fps	Great
A10G/A100	24-80GB	~30-60 fps	Full pipeline with all features

Recommended: NVIDIA GPU with ≥16GB VRAM for real-time processing.

Project Structure

racquet_sports_analyzer/
├── __init__.py              # Package exports
├── config.py                # All configuration (sport-specific params, model settings)
├── court_detector.py        # Court line detection + homography
├── player_detector.py       # YOLO11 + ByteTrack + pose keypoints
├── ball_tracker.py          # Classical CV + Kalman + TrackNet placeholder
├── rally_detector.py        # Rally segmentation from hit events
├── shot_classifier.py       # VideoMAE shot type classification + trainer
├── analytics.py             # Movement, shots, rally stats, biomechanics, playstyle
└── pipeline.py              # Main orchestrator (ties everything together)

run_analysis.py              # CLI entry point
requirements.txt             # Dependencies

Key Design Decisions

1. Modular pipeline over end-to-end: Each component can be upgraded independently. Start with classical CV ball tracking, upgrade to TOTNet later. Start with YOLO-pose, upgrade to ViTPose+ later.

2. Classical CV ball tracker as default: Works on all sports without training. The TrackNet-style deep model is a placeholder — train it on your data for 10x better ball detection.

3. Dual-signal hit detection: Fuses ball direction reversal (from ball tracker) with wrist velocity spikes (from pose keypoints). Either signal alone has false positives; combined they're robust.

4. Hit-frame-centered clips for shot classification: The BST paper proved this is significantly better than fixed-width temporal windows. We extract 3 frames before + 12 frames after each hit for classification.

5. Playstyle classification from multi-signal scoring: Movement speed + shot distribution + rally patterns → weighted score across aggressive/defensive/all-round/counter-puncher archetypes.

Datasets for Training

Dataset	Sport	Size	Where
ShuttleSet	Badminton	44 matches, 36K strokes	shuttleset.badminton.tw
TrackNetV2	Badminton	26 matches (ball annotations)	NCTU repo
BFMD	Badminton	19 matches, 16K hits	arxiv:2603.25533
BadmintonDB	Badminton	7.6K strokes, 8 matches	GitHub
TenniSet	Tennis	3.5K strokes, 5 matches	BST paper authors
WASB	Multi-sport	5 datasets	github.com/nttcom/WASB-SBDT

⚠️ Pickleball & Padel: No public academic datasets exist. You'll need to collect and label your own data. This is a competitive moat opportunity.

License

MIT

Citation

If you use this in your work:

@software{racquet_sports_analyzer,
  title={Racquet Sports Video Analyzer},
  year={2026},
  url={https://huggingface.co/Bot-Derpy/racquet-sports-analyzer}
}