The problem
In a patient exam room, doctors often miss subtle cues — a change in gait, difficulty standing up or sitting down, a shift in mood or affect — that don't get captured consistently from one visit to the next. For my senior capstone, our six-person team (spanning software, biomedical, and computer engineering) set out to build, from scratch, a smart exam room that could help surface these cues automatically and assist doctors with data most visits wouldn't otherwise capture.
My role
I was one of only two software engineers on the team and owned most of the software stack end to end — from the interfaces doctors and patients actually used, down to the computer-vision pipeline reading the room's cameras and microphones.
- Built the entire frontend in React, and a Flask backend using WebSockets and REST APIs to stream and process live sensor data in real time.
- Integrated the software with camera and microphone peripherals to track patient movement, including gait abnormalities and difficulty standing or sitting.
- Built a facial asymmetry detection pipeline from scratch, and helped train our own CNN in Keras for emotion detection — rather than relying on an off-the-shelf model.
- Added speech-to-text and sentiment analysis to capture patient mood and verbal cues alongside the visual signal.
- Built report generation using GPT to synthesize the compiled sensor and sentiment data into a clinician-facing report formatted to be compatible with common EHR systems, and the pipeline that persisted this data to AWS (DynamoDB + S3) to track a patient's progress over time.
Result
The system produced a base diagnosis and longitudinal progress reports that surfaced trends across visits and offered doctors smart recommendations grounded in data a single visit wouldn't otherwise capture. We presented the finished system to project sponsors, the Director of Software Engineering, and faculty reviewers.