Browser-Native Eye Strain Intervention

Icarus is a Chrome extension that detects eye fatigue in real time using MediaPipe's FaceLandmarker output, layered with a research-backed composite scoring formula. It runs entirely in-browser with no camera upload — all landmark processing happens locally on the client before structured metrics are posted to a backend.

The system uses a two-layer architecture. A Chrome Manifest V3 extension runs MediaPipe FaceLandmarker locally, processing webcam frames without transmitting raw video. The extension posts timestamped eye-metric samples to a Next.js backend every 10 seconds, which stores them in MongoDB Atlas and exposes an analytics API consumed by the dashboard.

This separation meant the extension could be lightweight and privacy-preserving while the backend handled aggregation, trend detection, and AI insight generation independently.

The core challenge was distinguishing genuine eye fatigue from normal blinking and squinting. A single metric — blink rate, for instance — produces too many false positives. The solution was a composite scoring formula combining three independent signals: eye squint and blink blendshapes from MediaPipe's FaceMesh output; Eye Aspect Ratio (EAR), a geometric metric derived from vertical and horizontal landmark distances; and HandLandmarker output detecting eye-rubbing gestures as a high-confidence fatigue indicator.

The three signals are weighted and evaluated across a 60-second sliding window, smoothing out transient spikes while remaining responsive to sustained fatigue patterns.

On the analytics side, the dashboard surfaces a 28-day GitHub-style strain heatmap populated from session data, and a Gemini 2.5 Flash integration that generates four personalised insight cards from weekly MongoDB metrics — cached for six hours to avoid redundant API calls.

Initial threshold calibration surfaced an unexpected problem: squint detection was triggering on normal concentration, not just fatigue. People naturally narrow their eyes when reading — the blendshape values were landing in the same range as early fatigue. Adjusting composite weights to reduce squint's contribution relative to EAR and hand-gesture signals brought false positives down significantly.

The Gemini integration required careful prompt engineering. Rather than passing raw MongoDB documents, I aggregated weekly metrics into structured summaries — session counts, average strain scores, time-of-day distributions — before composing a prompt that gave Gemini enough context to generate specific, non-generic recommendations.

The biggest operational pain point was ngrok's free-tier tunnel. The public URL invalidates on every restart, requiring an environment variable update and frontend redeploy each time the tunnel was cycled. This made demo-day prep brittle — worth noting for any future deployment moving to a stable backend host.

The full pipeline was live by demo day. Real session data flowed from the extension into MongoDB, the heatmap populated from actual use, and Gemini insights generated from compiled weekly metrics rather than mock data. The composite scoring formula held up across multiple testers without requiring per-user calibration.