mc3d — Monocular Visual SLAM for Minecraft
Reconstructs 3D point clouds from Minecraft gameplay video using ORB features, grid-based feature distribution, KLT tracking, and Ceres bundle adjustment.
A monocular visual SLAM system that reconstructs dense 3D point clouds from first-person Minecraft gameplay video. Detects ORB features across a grid-based distribution to ensure even coverage, matches descriptors with KLT optical flow fallback, triangulates points between keyframes, and refines geometry using Ceres bundle adjustment. Includes an automated hyperparameter tuner using Minecraft's known structure as a scoring signal.
- C++17
- OpenCV 4 (ORB, FAST, BFMatcher, KLT)
- Eigen 3 (linear algebra, geometry transforms)
- Ceres Solver (bundle adjustment, Huber loss)
- DBoW2 (place recognition, loop closure)
- CMake + vcpkg (Windows MinGW build)
- GLFW + GLAD (OpenGL viewer)
- Python 3 + Optuna (TPE Bayesian optimization)
- NumPy, SciPy, Matplotlib (scoring & visualization)
The system ingests either mp4 video clips or live DXGI screen capture, converts frames to grayscale, optionally applies CLAHE preprocessing to improve contrast in low-light scenes (caves, evenings), and detects ORB features. The core detection pipeline splits the image into a user-configurable grid (e.g., 4×4 cells) and enforces a minimum quota of features per cell to counteract ORB's tendency to cluster in high-contrast regions. When a cell returns fewer features than its quota, a fallback FAST detector with lower threshold fills in the sparse areas. Descriptors remain uniformly binary so downstream BFMatcher operates in a single pipeline.
After detection, the tracker maintains a set of 2D keypoints and 3D map points. For each new frame, it matches descriptors against the previous frame using BFMatcher with Lowe's ratio test; on match failure it falls back to KLT optical flow using Lucas-Kanade on the previously tracked point set. On bootstrap (the first two frames) it runs essential matrix recovery with RANSAC to estimate the camera's initial motion. Once tracking is initialized, new frames are locked to the map via 2D-3D PnP-RANSAC. When the camera has moved far enough or rotated sufficiently, a keyframe is inserted—its descriptors are triangulated against recent keyframes via epipolar geometry, and points are culled if they fail reprojection or depth bounds. Local bundle adjustment refines 15–25 recent keyframes and their observations using Ceres with a Huber loss.
The project includes a comprehensive automated tuner built on Optuna that optimizes roughly 25 hyperparameters (ORB feature count, FAST thresholds, KF insertion spacing, KLT ratio tests, reprojection error bounds, bundle adjustment window size, point culling rules, etc.) against a corpus of Minecraft clips. A scoring library evaluates each trial by measuring observation quality (penalizing transient noise), point yield (healthy geometry count), camera trajectory consistency (clip-specific motion priors), and axis-aligned plane snapping (Manhattan-world structure of Minecraft blocks). This coupling of realtime general-purpose SLAM with offline ground-truth calibration allows the system to converge on parameter values that maximize reconstruction accuracy without baking Minecraft-specific priors into the tracker itself.
- Grid-based feature detection that enforces minimum corner density per image region, solving ORB's clustering problem on uniform textures like dirt blocks.
- Dual-detector fallback: ORB detection augmented with FAST corners in sparse cells, then uniform binary descriptor computation for matcher compatibility.
- Multi-frame triangulation during keyframe insertion: matches new KF descriptors against the 4 most-recent older keyframes to catch close-range geometry that shifts dramatically between frames.
- Automated Bayesian hyperparameter tuner (Optuna TPE) that uses Minecraft's axis-aligned block structure as a free scoring signal, without leaking this prior into the realtime pipeline.
- Depth-aware point culling that exempts close-range geometry from stale-cull rules, preserving near-camera structure which naturally leaves the FOV in 3–5 frames as the player moves.