Technological Chromesthesia

The Concept

Sound has color

Chromesthesia is a neurological condition where sounds automatically trigger color experiences. This project translates that synesthetic phenomenon into software — using the physics of sound (frequency, amplitude) to algorithmically produce corresponding colors.

The mapping isn't arbitrary. Low bass frequencies map to deep reds and purples, mid-range tones map to greens and yellows, and high treble maps to blues and whites — following both human perceptual research and physical wavelength relationships between sound and light.

Core Idea

Frequency → Wavelength → Color

Sound waves have frequency (Hz). Light waves have wavelength (nm). The mapping between the two domains uses a logarithmic scale to match how humans perceive both — neither sound nor color is perceived linearly.

// Approximate mapping
20 Hz    → Deep Violet
80 Hz    → Indigo / Purple
250 Hz   → Blue
1,000 Hz → Green / Teal
3,500 Hz → Yellow-Green
8,000 Hz → Orange
20,000 Hz → Deep Red

Signal Pipeline

How It Works

From microphone input to rendered color — the full data flow through the system.

01

Audio Capture

Browser microphone captures raw audio via Web Audio API and sends PCM data chunks to the Flask backend via POST.

MediaRecorder · ArrayBuffer

02

FFT Frequency Analysis

NumPy and SciPy compute the Fast Fourier Transform of the incoming audio buffer — decomposing the signal into its frequency components and amplitudes.

numpy.fft · scipy.signal

03

Dominant Frequency Extraction

The system identifies the peak frequency and weighted average across frequency bands. Amplitude weighting prevents noise spikes from dominating the output.

argmax · weighted_mean

04

Frequency → HSL Mapping

The dominant frequency is mapped to an HSL color using a logarithmic scale across the audible spectrum. Pillow renders the resulting color as an image.

Pillow · HSL · log scale

05

SSE Stream to Browser

Flask pushes the color data to the browser as a Server-Sent Event. The frontend updates the background in real time — no WebSocket, no Socket.IO.

text/event-stream · EventSource

Frequency Band → Color Mapping

20–80 Hz

Sub Bass

80–300 Hz

Bass

300–1k Hz

Midrange

1–4k Hz

Presence

4–8k Hz

Brilliance

8–20k Hz

Air

Technical Decision

Why SSE Instead of Socket.IO

The original plan used Flask-SocketIO for real-time color updates — but a dependency conflict on the target environment forced a rethink. SSE turned out to be the better choice.

Flask-SocketIO Rejected

Required eventlet / gevent (conflicted with Pillow)
Heavy — full bidirectional WebSocket overhead
More complex client-side setup
Not needed — output is one-directional

Server-Sent Events Chosen

Pure HTTP — zero extra dependencies
Lightweight for one-directional data push
Browser-native EventSource API
Auto-reconnect on connection drop

          
          
          
        
stream.py — Flask SSE endpoint
from flask import Response, stream_with_context
import numpy as np
from scipy.io import wavfile

# Map dominant frequency to HSL color
def freq_to_hsl(freq_hz):
    lo, hi = 20, 20000
    clamped = max(lo, min(hi, freq_hz))
    hue = (1 - (np.log(clamped) - np.log(lo)) /
           (np.log(hi) - np.log(lo))) * 300
    return f"hsl({hue:.0f}, 85%, 55%)"

# SSE generator — yields color events
def color_stream(audio_data):
    freqs = np.fft.rfftfreq(len(audio_data), d=1/44100)
    spectrum = np.abs(np.fft.rfft(audio_data))
    dominant = freqs[np.argmax(spectrum)]
    color = freq_to_hsl(dominant)
    yield f"data: {color}\n\n"

# Flask route
@app.route('/stream')
def stream():
    return Response(
        stream_with_context(color_stream(audio_data)),
        mimetype='text/event-stream'
    )

Tech Stack

Built With

Flask

Web server · SSE routing

NumPy

FFT computation · array math

SciPy

Signal processing · filtering

Pillow

Color rendering · image output

SSE

Real-time push · EventSource

Web Audio API

Mic capture · PCM data

Results

What Was Achieved

<100ms

Audio-to-color latency via SSE

0

External realtime dependencies needed

6

Frequency bands mapped to color ranges

Live

Deployed and accessible via browser