SensorLLM

Aligning Large Language Models with Motion Sensors for Human Activity Recognition

EMNLP 2025 Main Conference

Zechen Li¹ Shohreh Deldari¹ Linyao Chen² Hao Xue¹ Flora D. Salim¹

¹ University of New South Wales, Sydney
² University of Tokyo

📢 News

05/2026: Excited to release our latest work AnyMo, a comprehensive framework for wearable motion understanding, covering synthetic IMU generation, geometry-aware pre-training, motion-language alignment, data resources, and a new benchmark.

05/2026: Excited to share ZARA — Zero-training Activity Reasoning Agents, a training-free, evidence-grounded LLM agent framework for motion time-series reasoning — accepted as an ACL 2026 Oral paper!

04/2026: Excited to share COMODO, a framework for Cross-Modal Video-to-IMU Distillation for efficient egocentric human activity recognition, accepted to IMWUT / UbiComp 2026!

🌟 Overview

SensorLLM is a two-stage framework that aligns sensor time series with human-intuitive text, enabling LLMs to interpret complex numerical data and achieve SOTA human activity recognition across varying sensor types, counts, and sequence lengths.

🔑 Key Features

Aligns sensor time-series with human-intuitive, annotation-free textual trend descriptions and summaries via a QA-based framework.
Sensor–Language Alignment Stage operates on single-channel, variable-length segments for fine-grained trend-text alignment.
Task-Aware Tuning Stage handles multi-channel, multi-sensor data for downstream human activity recognition (HAR).

📂 Datasets

The current implementation supports five HAR datasets: USC-HAD, UCI-HAR, MHealth, Capture-24, and PAMAP2.

To apply SensorLLM to other datasets, please refer to the code and configuration examples provided for the supported datasets. In particular, you may need to modify the corresponding entries in tsbackbone.yaml and adapt the data loading logic in the ./sensorllm/data folder to match your dataset’s format.

🚀 Getting started

Currently supported pretrained models:

- Time-series models: Chronos

- Language models: LLaMA
>
Other pretrained models can be used with minor modifications to the SensorLLM framework.

Sensor-Language QA Pairs Generation
We provide two example notebooks to generate QA pairs for aligning sensor time-series data with human-intuitive text:
mhealth_stage1.ipynb: Generates QA pairs for Stage 1 by aligning single-channel sensor segments with trend-based natural language descriptions.

mhealth_stage2.ipynb: Generates statistical information text for Stage 2, performing HAR classification using multi-channel sensor data.

You can also customize or extend the QA templates in these notebooks to generate more diverse types of sensor–language QA pairs for your own use cases.
Sensor–Language Alignment
To align sensor time-series data with text, run the following command:
torchrun --nprocpernode=[NUMGPUS] sensorllm/train/trainmem.py \ --modelnameorpath [LLMPATH] \ --ptencoderbackboneckpt [TSEMBEDDER_PATH] \ --tokenize_method 'StanNormalizeUniformBins' \ --dataset [DATASET_NAME] \ --datapath [TSTRAIN_PATH] \ --evaldatapath [TSEVALPATH] \ --qapath [QATRAIN_PATH] \ --evalqapath [QAEVALPATH] \ --outputdir [OUTPUTPATH] \ --modelmaxlength [MAX_LEN] \ --numtrainepochs [EPOCH] \ --perdevicetrainbatchsize [TRAIN_BATCH] \ --perdeviceevalbatchsize [EVAL_BATCH] \ --evaluation_strategy "steps" \ --save_strategy "steps" \ --savesteps [SAVESTEPS] \ --evalsteps [EVALSTEPS] \ --learning_rate 2e-3 \ --weight_decay 0.0 \ --warmup_ratio 0.03 \ --lrschedulertype "cosine" \ --logging_steps 1 \ --gradient_checkpointing True \ --savetotallimit 1 \ --bf16 True \ --fix_llm True \ --fixtsencoder True \ --model_type CasualLM \ --loadbestmodelatend True

Evaluation or Inference
To perform evaluation or inference for the Sensor–Language Alignment stage, run the following command:
python sensorllm/eval/eval.py \ --modelnameorpath [STAGE1MODEL_PATH] \ --ptencoderbackboneckpt [TSEMBEDDER_PATH] \ --torch_dtype bfloat16 \ --tokenize_method 'StanNormalizeUniformBins' \ --dataset [DATASET_NAME] \ --datapath [TSDATASET_PATH] \ --qapath [QADATASET_PATH] \ --outputfilename [OUTPUTFILENAME] \ --modelmaxlength [MAX_LEN] \ --shuffle False

Task-Aware Tuning
To perform a HAR task, use the following command:
torchrun --nprocpernode=[NUMGPUS] sensorllm/train/trainmem.py \ --modelnameorpath [STAGE1MODEL_PATH] \ --ptencoderbackboneckpt [TSEMBEDDER_PATH] \ --model_type "SequenceClassification" \ --numlabels [ACTIVITYNUM] \ --useweightedloss True \ --tokenize_method 'StanNormalizeUniformBins' \ --dataset [DATASET_NAME] \ --datapath [TSTRAIN_PATH] \ --evaldatapath [TSEVALPATH] \ --qapath [QATRAIN_PATH] \ --evalqapath [QAEVALPATH] \ --outputdir [OUTPUTPATH] \ --modelmaxlength [MAX_LEN] \ --numtrainepochs [EPOCH] \ --numtrainepochs [EPOCH] \ --perdevicetrainbatchsize [TRAIN_BATCH] \ --perdeviceevalbatchsize [EVAL_BATCH] \ --evaluation_strategy "steps" \ --save_strategy "steps" \ --savesteps [SAVESTEPS] \ --evalsteps [EVALSTEPS] \ --savetotallimit 1 \ --loadbestmodelatend True \ --learning_rate 2e-3 \ --weight_decay 0.0 \ --warmup_ratio 0.03 \ --lrschedulertype "cosine" \ --logging_steps 1 \ --bf16 True \ --fix_llm True \ --fixclshead False \ --fixtsencoder True \ --gradient_checkpointing True \ --metricforbestmodel "f1macro" \ --preprocess_type "smry+Q" \ --greaterisbetter True \ --stage_2 True \ --shuffle True
See ./sensorllm/data/utils.py for all available preprocesstype options or to make edits.

🌍 Citation

If you find this repository useful for your research, please cite our paper:

@inproceedings{li-etal-2025-sensorllm,
    title = "{S}ensor{LLM}: Aligning Large Language Models with Motion Sensors for Human Activity Recognition",
    author = "Li, Zechen  and
      Deldari, Shohreh  and
      Chen, Linyao  and
      Xue, Hao  and
      Salim, Flora D.",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    year = "2025",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.19/",
    pages = "354--379",
}