Developed an interactive real-time project on the Twitch gaming platform that enabled dynamic audience engagement through live streaming, leveraging NLP tooling to process and respond to user chat interactions.
- StackUnity, C#, Tensorflow
How it worked
A viewer's chat message moves through a single loop: it's read off Twitch chat, classified by a TensorFlow model into an intent and a room category, and handed to the Unity scene controller, which drops the matching piece of geometry into the shared build.
Fig. 01 — chat text becomes a rendered scene change in a single pass, with TensorFlow doing the only interpretation step in between.
What the demo shows
A reconstruction of the clip embedded on the original page: a chat rail feeding a live tally, and a cluster of low-poly geometry that grows as votes land.
Illustrative reconstruction based on the demo clip, not a frame capture — proportions and copy are approximate.
Six lenses on the loop
Read as a product rather than a demo reel: what job it does for a viewer, and where the design carries weight versus where it's simply implied by the stack.
01 · core loop
Type a command, see a number move, watch geometry appear. No page reload, no account: the entire interaction surface is the chat box a viewer already has open.
02 · users & motivation
Twitch chat is high-volume and low-attention by default. The bet: a visible, shared, cumulative effect (the house grows) turns passive lurking into repeated small actions.
03 · information architecture
A flat set of room categories keeps the vote tally legible at a glance, but caps expressiveness — no sub-choices for style, material, or placement.
04 · interaction model
Voting happens through free-text chat rather than buttons or polls. NLP is the interface, which is expressive but harder to guarantee correct.
05 · feedback signifiers
A percentage readout and new geometry are the only confirmations. There's no per-message acknowledgment, so a viewer can't easily tell if their own message landed.
06 · constraints from the stack
One NLP hop drives the scene directly, so a misclassification becomes a visible, public mistake — there's no moderation or confidence gate in the loop.
Where the design opens up
The tight, single-hop loop is exactly why it's hard to grow: NLP, game logic, and rendering all live in reach of one Unity process.
Problem 1 · coupling
Swapping or retraining the classifier means touching the same codebase that renders the scene — ML iteration speed is bound to game-build iteration speed.
Problem 2 · vocabulary ceiling
A flat category classifier can vote, but can't say "which wall" or "what material." Richer commands need a real grammar, not just a label.
Problem 3 · no safety valve
Nothing sits between "classified" and "rendered," so a misclassified or hostile message becomes a permanent, visible change to a shared scene.
A proposed three-layer extension
Splitting the single hop into a pipeline with a real boundary at each seam: streaming input owns the chat connection and rate control, Python + TensorFlow owns language understanding, and a dedicated C# modeling server owns the architectural rule set and mesh generation.
Fig. 02 — !build bedroom oak crosses three ownership boundaries; each arrow marks a payload-shape change, the seam the original design skipped by fusing NLP output directly to scene state.
From free text to architectural elements
The proposed grammar keeps the original's low floor — anyone can type !vote bedroom — while giving the classifier enough structure to target a specific element instead of a category tally.
| Chat message | Parsed intent | Scene effect |
|---|---|---|
!vote patio | VOTE · category=patio | Vote weight +1, no geometry change |
!build patio oak | BUILD · category=patio, material=oak | Oak-textured patio mesh instantiated |
!undo | UNDO | Last module removed, vote restored |
!lock roof 5m | LOCK · target=roof, duration=5m | Roof immutable for 5 min, throttles griefing |
nice roofline lol | below confidence threshold | Dropped — no scene effect |
What the extra layers buy
→ latency budget
Three network hops instead of one in-process call — target well under half a second, chat-to-render, so the "I typed it, it happened" feeling still holds live.
→ moderation surface
The grammar validator and per-user cooldown become the place to rate-limit spam and gate low-confidence parses — a seam the original single-hop design didn't have.
→ streamer controls
A separate modeling server can expose admin controls (pause, force-undo, reset) without touching the classifier or the Unity render client.
→ testability
Structured commands between layers two and three can be replayed from a fixture file, so the rule engine becomes testable without a live Twitch connection.
→ operational cost
Three deployable services instead of one Unity build — worth it once the classifier needs its own retrain/deploy cycle, not before.
→ what to measure
Command-to-render latency, parse confidence distribution, noise-vs-command ratio, and repeat participation per viewer session.