What five notebooks taught me about semantic communication

Turning a waveform into an image, losslessly

No FFT, no windowing — just a reshape. The trick is making that reshape mean something to a CNN.

Waveform vs. spectrogram

A waveform (amplitude vs. time) shows loudness, timing, silence — but hides frequency content by design. A spectrogram (time × frequency × energy) shows pitch, formants, harmonics. Most audio models train on a spectrogram for exactly that reason. This project doesn't use either directly — it does something narrower.

Framing here is a reshape, not an STFT

Classic "framing" chops audio into short overlapping windows for an FFT. This codebase's frame() does something much simpler: take a fixed W = 16384-sample clip and reshape it into a 128×128 grid, row-major, no windowing, no overlap, no frequency axis.

Row-major keeps each row temporally contiguous, so a 2-D CNN sliding a 5×5 filter over the grid is scanning real local structure — the same trick 2-D image models use, borrowed for audio. Column-major reshape would scatter same-row samples 128 apart in time, destroying that locality — it would look like structured noise to a convolution. W = 16384 was chosen specifically because it's a perfect square (128²) — a convenience, not a requirement.

Proving the encoder/decoder alone, channel off

If reconstruction fails with the easiest possible channel — none — nothing downstream can be trusted.

DeepSC_S is 115,177 parameters, spatial size 128×128 held constant throughout by "same" padding — only the channel-depth axis moves.

The SE-ResNet block

Each block computes x + s · t: a grouped convolution (cardinality 4, ResNeXt-style) produces t; a squeeze-excite branch — global-average-pool → 32→8→32 bottleneck → sigmoid — produces a per-channel gate s ∈ [0,1]³², recomputed fresh on every forward pass from the actual input (that's the "attention": data-dependent, not a static learned weight). The gated result is added back to the input as a residual.

Do the SE gates track anything real?

Zeroed the top-6 highest-gated channels vs. the bottom-6 lowest-gated channels and compared reconstruction damage: zeroing the important-looking channels hurt more (11.3 dB) than zeroing the ones the gate had already suppressed (13.0 dB). The gate isn't decorative — it's pointing at channels the network actually relies on.

Adding noise — and why power has to be pinned first

SNR is only a meaningful number if signal power is fixed. Otherwise "10 dB" means something different every run.

Power normalization

Before the channel, every batch of transmitted symbols is rescaled so mean|x_k|² = 1.0 — dividing by sqrt(measured_power), not the power itself, since power scales as amplitude². Two reasons this matters: real transmitters have a fixed energy budget, and SNR (signal power / noise power) is only comparable run-to-run if the numerator isn't drifting with random init and training step.

SNR sets the noise, not the signal

Signal power stays fixed at 1.0. What changes with SNR is noise std: σ = √(10^-SNR/10 / 2).

Training across a random SNR, every step

for epoch in range(1500):
    channel.snr_db = random.uniform(0, 20)   # fresh SNR every step
    x_hat = model(x, channel=channel)
    loss = MSELoss(x_hat, x)
    loss.backward()   # gradients flow through the noise op itself
    opt.step()

Sampling a new SNR every step — instead of fixing one — is data augmentation for noise level: it forces one encoding that's robust across the whole range, not tuned to a single point. torch.randn is differentiable, so gradients flow encoder ← channel ← decoder; the network learns to be robust to noise, not just to compress.

Fading: when the channel multiplies, not just adds

And the headline question — does a model trained for one channel survive a different one it never saw?

AWGN only adds noise: y = x + w. Real wireless channels also fade — multipath propagation makes the channel gain itself a random complex number: y = h·x + w.

• Rayleigh — no line-of-sight path, h ~ CN(0,1), pure scattering. Models dense-urban or indoor non-line-of-sight: deep fades are common.
• Rician — a line-of-sight path plus scatter, weighted by K-factor (LoS power / scattered power). K→∞ recovers AWGN; K=0 recovers Rayleigh.
• Perfect-CSI equalization — the receiver is assumed to know h exactly and divides it out (x̂ = y/h) before decoding, isolating fading's noise-amplification effect from the harder, out-of-scope problem of estimating h itself.

Train once, test on every channel

Two models, same architecture, each trained on one channel only (1,500 epochs, toy data, random SNR 0–20 dB) — then every model evaluated against all three channel types, no retraining.

Two things worth remembering here. First, the gap between matched and mismatched channels is largest at low SNR and nearly closes by 20 dB — noise dominates less, so the fading mismatch matters less. Second, and less obvious: the model trained on Rayleigh generalizes to AWGN better than the reverse. Training on the harder channel produces a more robust encoding than training on the easy one and hoping it transfers.

The headline result, on real speech

Same robustness question as stage 04 — now on the Edinburgh DataShare set, with a perceptual metric added.

Trained on a 2,000-clip subset of the 28-speaker clean set (of ~10k available — a deliberate, documented scope choice for a tractable single session, not a shortcut hiding a limitation), 40 epochs, AWGN only, then evaluated across all three channel types.

Same shape as the toy-data result in stage 04, but now with real absolute numbers: a ~4–5 dB gap between matched and mismatched channels at 0 dB SNR, narrowing to under 1 dB by 20 dB SNR.

PESQ — a metric SDR can't give you

SDR is a pure signal-energy ratio; it can't tell you what actually sounds bad to a human ear. PESQ (Perceptual Evaluation of Speech Quality, scale 1.0 worst – 4.5 best) is the industry-standard perceptual metric for speech codecs, wired in as an optional dependency that degrades gracefully to an SDR-only plot if the pesq package isn't installed. Here it tracked the same channel ordering as SDR (awgn best, rayleigh/rician trailing) but stayed low in absolute terms, ~1.0–1.35 — the expected signature of a model trained on a 2k-clip subset for 40 epochs rather than the paper's full-scale run, not a bug.

Two small engineering notes

• Checkpoints are saved every N_EPOCHS/4 epochs during training to build a "quality over time" gallery — the same held-out clip decoded with progressively later weights.
• A later revision made training skip itself entirely if checkpoints/deepsc_s_awgn_final.pt already exists — no reason to burn a real-data training run again just because the notebook restarted.