Continuous Fingerspelling Dataset for Indian Sign Language

WSLP @ AACL-IJCNLP 2025

R Kirandevraj¹, Vinod K Kurmi², Vinay P Namboodiri³, CV Jawahar¹

¹IIIT Hyderabad, India ²IISER Bhopal, India ³University of Bath, UK

Dataset Paper (Coming Soon) Code (Coming Soon)

We introduce the continuous fingerspelling dataset for Indian Sign Language, comprising 1,308 video segments with aligned text annotations. The dataset captures authentic coarticulation patterns from professional signers, supporting research in fingerspelling recognition and sign language processing.

Fingerspelling: "sulochana das"

Fingerspelling: "vinesh phogat"

Abstract

Fingerspelling enables signers to represent proper nouns and technical terms letter-by-letter using manual alphabets, yet remains severely under-resourced for Indian Sign Language (ISL). We present the first continuous fingerspelling dataset for ISL, extracted from the ISH News YouTube channel in which fingerspelling is accompanied by synchronized on-screen text cues. The dataset comprises 1,308 segments from 499 videos, totaling 70.85 minutes and 14,814 characters, with aligned video-text pairs capturing authentic coarticulation patterns. We validate dataset quality through annotation by a proficient ISL interpreter, achieving a 90.67% exact match rate for 150 samples. We further establish baseline recognition benchmarks using a ByT5-small encoder-decoder model, which attains 82.91% Character Error Rate after fine-tuning. This resource supports multiple downstream tasks including fingerspelling transcription, temporal localization, and sign generation.

Dataset Overview

1,308 Video Segments

70 min Total Duration

14,814 Characters

499 Source Videos

3 Signers

90.67% Validation Acc.

Character Distribution

92.64% Alphabets (13,724 characters)
6.82% Spaces (1,011 characters)
0.17% Numbers (25 characters)
0.36% Other characters (54 characters)

Dataset Structure

ISL-Fingerspelling/
├── videos/                        # 1,308 MP4 video files
├── fingerspelling_annotations.csv # Segment annotations
├── localization_timestamps.csv    # Temporal localization in source videos
└── README.md

fingerspelling_annotations.csv

Maps video segments to their transcriptions:

uid: Unique segment ID (format: {video_id}_seg{index})
text: Fingerspelled text

localization_timestamps.csv

Contains temporal boundaries of fingerspelling segments in the original YouTube videos:

video_id: YouTube video ID
segment_index: Segment index within the video
start_time, start_sec: Start timestamp and seconds
end_time, end_sec: End timestamp and seconds
duration_str, duration_sec: Duration in timestamp and seconds
transcript: Fingerspelled text

Citation

If you use this dataset in your research, please cite:

@inproceedings{kirandevraj2025islfingerspelling,
  title={Continuous Fingerspelling Dataset for Indian Sign Language},
  author={Kirandevraj, R and Kurmi, Vinod K and Namboodiri, Vinay P and Jawahar, CV},
  booktitle={Proceedings of the Workshop on Sign Language Processing (WSLP) at AACL-IJCNLP},
  year={2025}
}

Acknowledgments

Data sourced from publicly available ISH News YouTube videos. 407 of 499 videos overlap with the iSign dataset. This dataset is released under CC BY-NC 4.0 license for research purposes only.