FitSense
Mobile app that uses a CNN-LSTM model to recognize physical exercises from motion sensor data in real time, with a human feedback loop that turns user corrections into training data.
Overview
FitSense is a mobile application that automatically recognizes physical exercises in real time using motion sensor data from the device's accelerometer and gyroscope. Users simply start a workout session and the app predicts what exercise they are performing, no manual logging required.
The core of the system is a hybrid 1D CNN-LSTM model trained on the mHealth dataset, capable of classifying 12+ exercise types. What makes FitSense distinctive is its human-in-the-loop feedback system: when the model gets a prediction wrong, users can correct it directly in the app, and that correction becomes labeled training data for future model improvements.
The ML System
The exercise classifier combines two complementary architectures:
- CNN layers extract local temporal patterns from raw sensor signal windows
- LSTM layers capture longer-range sequential dependencies across the time series
- Trained on the mHealth dataset, a public benchmark for activity recognition from body-worn sensors
- Model is served via a Firebase Cloud Run API, decoupling inference from the mobile client
Human Feedback Loop
One of the core design decisions in FitSense is treating user corrections as first-class data. When the model predicts the wrong exercise, the correction is:
- Stored in Firestore alongside the original sensor data
- Used as labeled training data to retrain and improve the model in future iterations
This creates a continuous improvement cycle, the more the app is used, the more accurate it becomes. It also addresses one of the key challenges in activity recognition: variability in sensor patterns across different devices and users.
My Contribution
This was a solo project, giving me full ownership of every layer of the system:
- ML Model: Designed, trained, and evaluated the CNN-LSTM architecture
- Backend: Built and deployed the inference API on Firebase Cloud Run
- Mobile Frontend: Built the full React Native app including session management, live prediction display, and the feedback UI
- Data Pipeline: Handled sensor data preprocessing and model export
Technical Stack
- Mobile: React Native (Expo)
- ML Framework: PyTorch
- Backend: Firebase (Cloud Run, Firestore, Authentication, Cloud Storage)
- Dataset: mHealth Dataset
- Language: Python (model), TypeScript (frontend)
Challenges & Learning
Building an end-to-end ML system from sensor to prediction taught me:
- Time series modeling: Working with raw sensor signals requires careful windowing, normalization, and architecture choices that differ significantly from typical ML tasks
- ML deployment: Exporting a PyTorch model and serving it reliably via a cloud API involves tradeoffs around latency, cold starts, and model versioning
- Feedback-driven design: Designing systems where user behavior generates training data requires thinking carefully about data quality and label reliability
- Full-stack ownership: Owning every layer, from model training to mobile UI, gives you a much deeper understanding of where systems break down