The Mouse Action Recognition System (MARS) is a deep learning based system for automated pose estimation and social behavior classification in pairs of interacting mice. It combines a user-friendly graphical user interface with fast and powerful computer vision and machine learning algorithms.

MARS's novel approach to extracting features from the animals' poses and trajectories allows for highly accurate classification of social behavior. MARS's pose estimator was trained on manual keypoint annotations of 15,000 video frames; all keypoints on each frame were labeled by five human annotators to ensure high training set quality. The MARS behavior classifiers were developed and tested on a novel dataset of videos of interacting pairs of mice, comprised of over 14 hours of annotated videos recorded at high resolution. Both datasets will be made publicly available for benchmarking of other pose and behavior classification systems.

MARS is an open-source sofware developed by the laboratories of Pietro Perona and David Anderson at Caltech.


Detection and pose of each mouse, maintaining the identity across the video frames.

Mouse tracker and behavior classifier.


If you make use of MARS or the MARS training data, please cite the following reference in any publications:
The Mouse Action Recognition System (MARS): a software pipeline for automated analysis of social behaviors in mice.
Cristina Segalin, Jalani Williams, Tomomi Karigo, May Hui, Moriel Zelikowsky, Jennifer J. Sun, Pietro Perona, David J. Anderson, and Ann Kennedy.
pdf | bibtex


  • 06/22/2020 Migration to GitHub in preparation for release.


For question about the software and the dataset please contact Ann Kennedy [kennedya[at]] or Cristina Segalin [segalin.cristina[at]].