Research Projects | 2026
Between the Frameworks
A research case study on teaching AI to see what isn't there: the shadow.
Most AI today is built to find the average. Trained on vast oceans of data, these tools are designed to seek out the most common outputs, smoothing over the messy, complex reality of a world that rarely exists in clear ones and zeros. Fearful that out-of-the-box models can never close every gap or understand every unique context, and as an artist who deeply values cultural richness and varied human experience, the idea of a homogenized, “one-size-fits-all” future is alarming.
This project began as a way to explore that tension through a deceptively simple metaphor: the shadow.
Shadows exist in a gray zone, they shift over time, lack clear boundaries, and are notoriously difficult for AI to label. Traditional computer vision tools miss the broader context of the scene, while massive global AI models lack the sensitivity to truly understand what they’re looking at. It created a perfect problem space where both old and new methods fell short.
Instead of relying on monolithic models that force everything toward a single consensus, this project proposes a different path forward. The future of AI isn’t one giant brain, but an ecosystem of small, distilled models working together in an agent-to-agent workflow, much like object-oriented code, where a global model calls upon hyper-specialized tools at the exact moment they’re needed.
By injecting diversity at the point of need, we can build AI that is dynamic, adaptable, and capable of learning nuances far beyond its initial training.
Why Traditional Computer Vision Fails
Most classical Computer Vision (CV) tools rely on a simple assumption: the object you want to find looks distinctly different from its background. A cat has a different color and texture than a couch, no matter where you photograph it.
Shadows break all of these standard rules:
Brightness Thresholds Fail
A shadow on a brightly lit white wall can actually be brighter than an unshaded patch of dark brick right next to it. Simple light-versus-dark cutoffs fall apart immediately.
Color Thresholds Fail
Shadows depending on which surface they are displayed on will vary. Over time their color will change and they dont seperate uniformally from the surface which they apear on
Edge Detection Fails
The Real Task: Isolating the Tree’s Cast Shadow
Most tools assume “shadow” is a single global category. In practice, a physical scene contains three distinct, overlapping lighting signals:
• Target Cast Shadow: The specific, high-frequency boundary cast by the subject (e.g., the tree branch).
• Surface/Wall Shading: The inherent gradient or texture of the physical receiver plane.
• Ambient Occlusion: The broad, low-frequency darkening from distant buildings, overhangs, or sky cover.
Generic segmentation tools blur these together or select the entire dark region of the wall. The task here is surgical: isolating the specific cast shadow while ignoring the ambient wall gradient sitting directly beneath it.
How Modern AI Tools Struggle with the Shadow
Segment Anything Model (SAM):
Looking into SAM’s internal attention maps reveals a fundamental structural blind spot: SAM is trained to look for objects, not phenomena.
The Ice Cube Success: SAM excels at segmenting an ice cube because an ice cube is a discrete physical subject with a defined boundary, distinct texture, and clear visual presence.
The Shadow Failure: A shadow is an optical phenomenon painted onto a surface, not an object itself. Because “wall” is a heavily reinforced object class in SAM’s training weights, its attention mechanism
Generative Models (Diffusion & Vision Transformers):
Building The Frame
Understanding why standard AI and computer vision fails at shadows isolation is only half the battle. The real work began in deciding how to fix it.
I don’t have the resources for a single massive, brute-force model which demands enormous amounts of data, computing power, and time. Instead, I approached this as an architectural design problem. My feeling was I could combine the fast precision of traditional algorithms with the more global context of modern deep learning.
What followed was an iterative process of experimentation. I started by pushing custom non-AI pipelines to their limits, mapped out their failure points, and then progressively introduced lightweight, specialized neural networks. Each attempt built upon the lessons of the last, gradually moving toward a modular, distilled system capable of seeing both the tiny pixel details and the bigger picture.
Step 1: Set A baseline using only the foundations.
This system was built without machine learning or neural networks, relying instead on a fundamental rule of optics: a shadow requires both a tonal drop and a corresponding change over time.
Each of the seven tools analyzed the scene across different spatial scales and windows of time, contributing weighted votes toward a maximum possible score of 38 per pixel:
- Layer 1 Output (Boundary Lock – Tools A & E):
- Layer 2 Output (Penumbra & Fill – Tools B & D):
- Layer 3 Output (Deep Interior – Tool F):
- Layers 4 & 5 Output (Sharpening & Gap Sealing – Tools C & G):
- Final Cascade Consensus (30% Threshold):
What was reassuring is that the entire pipeline was built on pure physics and temporal rules—no manual training, no machine learning weights, and no neural networks.
For an unsupervised, rule-based system built from scratch in two days, this was a massive win. It proved that classical computer vision could capture complex, moving physical structures far better than off-the-shelf tools like GrabCut—setting a strong benchmark for the deep learning models that followed.
Step 2: Transferring Physics to a Lightweight CNN
Focusing on small applications of AI with low power use, speed and performance as criteria, I built a hyper-efficient, 3-layer Convolutional Neural Network (CNN) with only 6,129 total parameters. Instead of forcing the network to learn physics from scratch, I fed it a custom 10-channel physics input stack, baking global context, motion, and color logic directly into the inputs before the CNN ever processed a single pixel.
How the Sliding Window Works
Instead of judging each pixel in isolation, the CNN acts as a sliding window across the image:
- Every pixel decision is informed by a 5×5-pixel neighborhood, allowing the network to analyze surrounding spatial patterns.
- Layer 1 reads the 10 physics channels directly to generate 16 pattern maps.
- Layer 2 looks for “patterns within patterns,” expanding to 32 feature channels.
To keep inference fast, the network runs at half-resolution and upscales the final output. During training, I applied 5x edge weighting to the loss function, forcing this tiny 6k-parameter network to focus obsessively on getting the tricky shadow boundaries right.
CNN: Archetectural overview
Channels 1–4: Spatial Edges & High-Frequency Gradients
Channels 5–8: CIELab Color Deltas & Luminance Shift
Channels 9–10: Multi-Frame Temporal Motion History
Layer one reads the ten physics channels directly and produces sixteen new pattern-channels. Layer two takes those and produces thirty-two more. Layer three collapses all thirty-two down to a single number per pixel: how likely is this one pixel to be shadow.
Step 3: Elevating Scene Context with DINOv3 & Self-Supervised Learning
After proving what classical physics rules could do, the next breakthrough meant stepping into modern vision architectures. To give the system true “global awareness”—the ability to understand the whole scene rather than just localized pixels—I integrated DINOv3, a state-of-the-art Vision Transformer (ViT) model. this model is free and can fit on the averaege consumer desktop.
What Makes DINOv3 Different?
Self-Supervised Learning: The model trains itself by looking at millions of unlabeled images, learning the deep structural and geometric relationships of the physical world.
Student-Teacher Distillation: A larger “teacher” network observes an entire image and guides a smaller “student” network to recognize the same underlying structures from cropped or masked views. Over time, the student becomes hyper-efficient at understanding scene context without needing giant datasets.
Attention Routing: the attention maps after fused are then sent though a router where the highest performing values are passed in. The router is a small nureal network design to look at different patterns and build a probability indexed based on the individual layers weights. The router is trained against a few hand drawn truhts
The Breakthrough: Coarse-to-Fine Fusion
Even with a powerful vision transformer, extracting razor-thin shadow boundaries required a smart sampling strategy. I designed a two-stage Coarse-to-Fine workflow:
The Coarse Pass (The Generalist): Takes a quick, broad look at the entire photo to map out the general shadow boundaries and scene layout.
The Fine Pass (The Specialist): Zooms in only on the fuzzy, high-uncertainty edge crops to evaluate intricate details.
The Fusion Layer: Weighs the coarse prediction against the fine detail prediction, choosing the most reliable score for every pixel.
The Findings: Why the Generalist Must Lead
Testing the sampling passes independently revealed a critical lesson in AI
architecture: specialized models fail when they lose big-picture context.
Does the choice of specialist change with the light? Yes.
The channel you’d assume is safe to cut isn’t
Final weight measures output contribution, not necessity
Given sobel’s near-zero final blend weight, the obvious next move is to drop it and simplify the architecture. Tried directly: removing sobel cost real accuracy, despite it barely contributing to the final output.
The explanation is cross-channel self-attention: before any channel’s output gets blended by its tiny final weight, every channel already attended to every other channel and adjusted its own answer using that context. Sobel’s job wasn’t to contribute much of its own signal at the end — it was to calibrate the other channels earlier in the pipeline.
What was Captured
The concrete results, the breakthrough moments, and the unexpected strengths of the model.
Single-Shot Transfer & Focal Depth Intelligence
Without any additional training, the model was tested on a completely new scene—a different tree cast onto a similar wall at a different distance:
In-Focus Shadow Identified: The model crisp-segmented the foreground tree branch shadow sitting directly against the wall.
Background Layer Excluded: Crucially, it completely ignored a secondary, blurry shadow cast by a distant tree further down the wall.
Traditional computer vision tools would have grouped both dark regions together solely based on brightness. The CNN demonstrated a level of focal depth awareness, isolating the primary subject while suppressing layered background noise in a single shot.