The Mouse Action Recognition System (MARS) is a deep learning based system for automated pose estimation and social behavior classification. It combines a trio of tools:

  • MARS itself is a Python-based pipeline for out-of-the-box pose estimation and classification of social behavior in videos of interacting mice, designed for experiments conducted in our standardized behavior acquisition hardware. It combines a user-friendly graphical interface with powerful computer vision and machine learning algorithms developed and tested on a novel dataset of videos of interacting pairs of mice. MARS comes pre-packaged with a pose estimator trained on manual keypoint annotations of 15,000 video frames of interacting mice, and a set of behavior classifiers trained to detect aggression, mounting, and close investigation behaviors.
  • MARS_Developer is a Python-based tool for re-training MARS pose models or behavior classifiers on new datasets. Use it to create your own custom tracking/behavior classification tools, that you can then deploy in the MARS interface.
  • the Behavior Ensemble and Neural Trajectory Observatory (Bento) is a Matlab-based tool for managing, visualizing, and analyzing multimodal experimental datasets, including neural recording, video, behavior annotation, and audio data. The BENTO user interface allows you to see neural activity and behavior side-by-side, and supports exploratory data analysis of neural activity using methods like PCA, clustering, and event-triggered-averaging, without requiring any coding ability.

The MARS training datasets

MARS was trained using 15,000 video frames manually annotated for animal pose, and 14 hours of video manually annotated for multiple social behaviors of interest. To quantify inter-annotator variability in behavior identification, we also collected manual annotations of social behaviors from eight trained individuals on a collection of 10 videos (over 1.5 hours) of interacting mice.

We have made all three datasets publicly available under a Creative Commons noncommercial license CC BY-NC 4.0.

The datasets are hosted by the Caltech library, and can be downloaded from the Datasets page.


If you make use of MARS or the MARS training data, please cite the following reference in any publications:
The Mouse Action Recognition System (MARS): a software pipeline for automated analysis of social behaviors in mice. (2020) Segalin, C., Williams, J., Karigo, T., Hui, M., Zelikowsky, M., Sun, J.J., Perona, P., Anderson, D.J., and Kennedy, A. bioRxiv 2020.07.26.222299
link | bibtex


  • 09/20/2021 Site updated to include the new MARS_Developer repo, plus new dataset links.
  • 06/22/2020 Migration to GitHub in preparation for release.


For question about the software and the dataset please contact Ann Kennedy [ann.kennedy[at]] or Cristina Segalin [segalin.cristina[at]].