A behavior classifier for freely interacting mice, trained on annotated video from 15 laboratories. HiDRA scores 33 social and solitary behaviors from pose tracking, and it works on data from labs it never saw during training.
assets/video/demo-ethogram.mp4 (see VIDEOS.md).
Supervised behavior classifiers such as MARS, SimBA, JAABA and DeepEthogram can match human annotators when they have enough training examples from the setup they are used on. Each one is trained on a single lab's videos and annotations, and it does not generalize to new experimental settings. The field has ended up with one-off tools that stay in the lab that built them, and behavior scored in one lab cannot be compared directly with behavior scored in another.
HiDRA was trained on DooM, a dataset assembled from previously unshared experiments in 15 labs. Training on that variety produced a shared representation of mouse movement. Classifier heads for the same behavior, trained on different labs' annotations, converge on similar weights, and they transfer to new datasets without retraining.
assets/video/cross-lab.mp4 (see VIDEOS.md).
| Property | Single-lab classifiers | HiDRA |
|---|---|---|
| Training data | One lab's annotated videos | 180 hours of annotated video from 15 labs, with different arenas, cameras, strains and pose skeletons |
| Features | Hand-engineered pose features (distances, angles, velocities) or raw pixels | Learned by graph neural networks trained, without labels, to predict how each keypoint moves next |
| Behaviors | Whatever the training lab annotated | 33 behaviors as 82 classifiers, each one a specific lab's definition, all available at once |
| A new lab or setup | Annotate new data and retrain | Run as is. If annotations are off, recalibrate a threshold or fine-tune one head on a few annotated clips |
| Adding a behavior | Annotate and train a new classifier | Fine-tune a new 258-parameter head in under 10 minutes on one GPU, without affecting performance of the remaining classifiers |
| Acquisition hardware | Tied to the training lab's camera and tracking setup | Accuracy is largely independent of frame rate, video resolution and number of tracked keypoints |
The 15 labs in DooM never shared a scoring manual. Between them they used 121 different names for behaviors, which we merged into 33 distinct behaviors: thirteen scored by only one lab, the rest by two or more labs each working from its own operational definition.
Importantly, HiDRA does not average definitions into a consensus classifier. Instead, it trains a separate head for each lab that scored a behavior, which gives 82 heads. Attack, for example, has eight: one for each lab that annotated it.
That makes the heads useful in two ways. You can pick the head whose definition is closest to your own lab's scoring. You can also run several heads on the same video to see how different groups of annotators would have scored it. On CalMS21, six labs' attack and sniff classifiers agree on most bouts but differ in how long attack bouts run and in where attack ends and sniff begins.
DooM contains more than 600 top-view videos of interacting mice from 15 labs, with pose tracking, metadata and frame-by-frame manual annotation. All of it comes from published or ongoing experiments, so it reflects real variation: resolution, session length, frame rate and arena size each vary by at least an order of magnitude.
We ran a two-month Kaggle challenge on DooM that drew over 30,000 submissions from 1,781 participants in 85 countries. The winning entry learned its features with self-supervised graph neural networks instead of hand-crafting them. HiDRA is a lightly modified version of that model, retrained on a refined version of the dataset.
Browse each lab's recording setup and the behaviors it annotated, or pick a behavior to compare how different labs defined it. Definitions are quoted from the labs' own documentation.
HiDRA would not exist without the 15 laboratories that shared their data. Each contributed previously unshared video, pose tracking and frame-by-frame behavior annotations from its own experiments. Together these became the DooM dataset, which we are releasing openly alongside HiDRA.
The model began as a community effort too. It builds on the winning entry of the DooM Kaggle challenge, and several of the challenge's top-ranked participants are members of the consortium.
We thank Addison Howard, Sohier Dane and Elizabeth Park at Kaggle for their support in running the Days of our Mice competition.
HiDRA consists of a core shared network that learns the dynamics of mouse movement, with each behavior classifier a simple threshold-linear readout at the end.
2D keypoints for up to four mice, from SLEAP or any other tracker. Skeletons differ between labs, so HiDRA uses whichever of its core body parts are present and treats the rest as missing.
Graph neural networks over single animals and animal pairs learn to predict each keypoint's next position. This stage needs no behavior labels.
Features are merged for each actor-target pair of mice, then passed through recurrent temporal filtering layers conditioned on a lab embedding.
Each (lab, behavior) pair gets its own linear readout. You choose the "attack" whose definition matches your lab's convention.
The released model is an ensemble of five configurations that differ in temporal resolution and number of keypoints. Their predictions are averaged.
On the held-out DooM test set, HiDRA reaches a per-lab macro-averaged F1 score of 0.66 across all 82 classifiers. Predicted time spent in each behavior closely tracks human scoring.
For about half of the classifiers, most remaining errors are disagreements about exactly when a bout starts, within half a second of the human-annotated onset. Human annotators are inconsistent about onsets too, and HiDRA applies the same rule every time.
We regressed the F1 scores of all 82 classifiers against properties of their data. The amount of training data and the rarity of the behavior predicted performance. Frame rate, video resolution and the number of tracked keypoints did not.
We also swapped HiDRA's pose features for features from VideoPrism, a video foundation model. The pose-based model was better for 80 of 82 behaviors. A lab does not need a high-speed camera, a dense skeleton or a large video model to get accurate classification.
Deep learnimg models are highly flexible: a model trained on 15 labs could have learned 15 separate, lab-specific representations of animal movement. Instead, HiDRA heads for the same behavior trained on different labs' annotations are often more similar to each other than to other heads from their own lab.
This shared representation allows HiDRA classifiers to be transferred between labs, and also gives a way to compare how labs define the same behavior.
CalMS21 is a public benchmark whose behavior annotations HiDRA never saw. Applied to it, HiDRA's attack, sniff and mount classifiers agreed closely with the held-out human annotations and scored close to the top model from the original CalMS21 benchmark, which had been trained on those annotations. The HiDRA classifiers were trained by different annotators, on different videos, in different labs.
In leave-one-lab-out tests, home-cage setups with typical pose skeletons came close to the full model's performance. Setups unique to one lab, such as four-mouse arenas or unusual skeletons, scored lower and recovered with fine-tuning.
We tested fine-tuning of HiDRA classifiers on a new three-mouse home-cage dataset not seen during training. The pretrained self-groom and sniff-genital classifiers already performed well with no tuning, and fine-tuning only the output head on two annotated clips took under 10 minutes on a single GPU and improved both. The same procedure turned the head for climbing, which detected jumping down at near-chance level (F1 = 0.07), into an accurate jump-down classifier.
In human-in-the-loop annotation, reviewing HiDRA's candidate bouts replaced searching the video for them and cut annotator time 1.5- to 2.9-fold. HiDRA also tolerates poor pose estimates: with SLEAP models trained on only 100 labeled frames, attack and sniff classification had nearly reached its plateau (500 frames for mounting), while pose accuracy itself kept improving up to 10,000 frames.
We applied all of HiDRA's classifiers, unchanged, to published resident-intruder recordings of four mouse lines carrying autism-associated mutations. HiDRA reproduced the previously reported phenotypes and found differences in other behaviors. Transition analysis showed why Nlgn3 heterozygotes move from attack to approach more often: after a fight they are more likely to separate from their partner and then re-engage.
Decoders that combined HiDRA bout statistics with behavior-conditioned movement features predicted genotype as well as decoders built on keypoint-MoSeq syllables. HiDRA needs no model fitting for this, and each feature is a named behavior.
In a second dataset, 41 male mice met a male or female intruder on three consecutive days. HiDRA matched manual scoring of how behavior changed with experience, including the distinction between mounting with and without intromission. Across the full set of classifiers it found a broad shift in resident behavior over the three days that depended on the sex of the intruder.
HiDRA is a Python package. Weights (about 660 MB) download from Hugging Face on first use.
pip install \ 'hidra[torch] @ git+https://github.com/neuroethology/HiDRA' hidra-download-models # fetch weights into models/ hidra-predict --list-heads # the 82 (lab, behavior) heads # run two heads on a folder of pose files hidra-predict tracking/ --out results/ \ --pix-per-cm 16 --fps 30 \ --labs GroovyShrew --actions rear,sniffall
import hidra hidra.heads(action="attack") # which labs have an attack head hidra.predict("tracking/", out="results/", labs=["LyricalHare"], actions=["attack"], pix_per_cm=16, fps=30) bouts = hidra.bouts("results/") # ethogram as a DataFrame
Each input file produces <name>.bouts.csv (thresholded bouts with subject, target, lab and behavior) and <name>.frames.parquet (per-frame probabilities).
video_frame, mouse_id, bodypart, x, y) or SLEAP .slp files. Up to four mice.No annotation needed. Pick the heads whose definitions match yours and run them on your videos.
Uses a few annotated clips. Refit a decision threshold, or retrain one head on your arena and annotation style.
Start from the closest existing head and train a new column without touching the rest.
Rebuild the self-supervised trunk or the supervised layers with the research code. Takes GPU-days.
@article{marks2026community,
title = {A community benchmark enables high-resolution, cross-lab quantification of mouse social behavior},
author = {Marks, Markus and Yang, Ruize and Liu, Ziang and Ye, Ryan and Scaria, Lara K. and Golden, Sam A. and Sun, Jennifer J. and Pereira, Talmo and Kennedy, Ann and {The Days of our Mice Consortium}},
journal = {bioRxiv},
year = {2026},
month = oct,
note = {Preprint, posted October 20, 2026},
doi = {10.64898/2026.10.18.758876},
url = {https://www.biorxiv.org/content/10.64898/2026.10.18.758876v1}
}
}