Flutist Cue Dataset

This repository contains data used in the study “Predicting Flutist Onset Timing in Duet Performance: A Multimodal Analysis of Gesture and Breath Cues.” The dataset captures how flutists use gestures and breaths to synchronize with a pianist when starting a piece, providing a multimodal perspective on cue-based ensemble performance.

Folder Structure

flutist_cue_dataset/
├─ annotation/
├─ motion_numpy/
├─ motion_pkl/
├─ motion_png/
├─ sheetmusic/
└─ wav/

0. Filename Explanation

This dataset uses filenames such as P4_S2_T3_13 to encode four pieces of information, interpreted as follows:

• P4: Participant (flutist) number 4
• S2: The second set of sheet music (i.e., from folder S2)
• T3: The third trial (or attempt)
• 13: The 13th piece within that sheet-music set

Hence, a file named P4_S2_T3_13 indicates “Participant #4, using the second sheet-music set, on their third attempt, performing piece number 13.”

1. annotation/

• Contents: A single CSV file containing six manually annotated time markers per trial:
  1. Start (experimenter’s clap to align all signals),
  2. Breath Onset (start of the flutist’s inhale),
  3. Breath Offset (end of the inhale),
  4. Flute Onset (the flutist’s actual note onset),
  5. Piano Onset (the pianist’s note onset),
  6. 2nd Measure (the flutist’s second measure start).
• Details:
  • Each row corresponds to a single trial from one flutist (P1–P20).
  • Timestamps are typically in seconds.

2. motion_numpy/

• Contents: NumPy arrays (.npy files) with processed gesture-cue motion curve information for each flutist (P1–P20).
• Details:
  • Each file includes an array that stores frame indices for max/min peaks of position, velocity, and acceleration, plus relevant event frames:

    points_array = np.array([
        position_max_peak,
        position_min_peak,
        velocity_max_peak,
        velocity_min_peak,
        acceleration_max_peak,
        acceleration_min_peak,
        flute_onset,  
        second_measure,       
        pre_second_measure    # second measure by -1 from flute onset
    ])

3. motion_pkl/

• Contents: Python pickle (.pkl) files containing raw or less-processed gesture motion data for each flutist (P1–P20).
• Details:
  • These typically contain full optical-flow curves used for further processing or re-analysis.
  • Good for replicating or extending the pipeline, e.g., re-detecting peaks or applying custom filters.

4. motion_png/

This folder contains PNG snapshots of the motion curves (position, velocity, acceleration) for each trial, illustrating how the system detected or failed to detect gesture peaks. The subfolders categorize trials based on detection or annotation status:

1. 1_automated

   • Trials in which automatic detection of six (pos, vel, acc) max/min peaks succeeded.
   • No manual intervention was required.

2. 2_annotated

   • Trials that required manual annotation of peaks.
   • Automatic detection was incomplete or imprecise, so peaks were identified/confirmed by human inspection.

3. 3_no_peak

   • Trials where no valid peak could be found, even after manual checking.
   • Indicates that the gesture might not have shown clear max/min in the expected time window.

4. 4_motion_detection_error

   • Trials where motion curve extraction itself failed or was highly inaccurate.
   • May result from tracking failure, camera occlusion, or excessive motion blur.

Each PNG file typically shows the derived motion curves (position, velocity, acceleration) over time, with detected peaks (if any) marked. These visual references help verify peak locations and diagnose motion-tracking or annotation issues.

Note on Filenames
In some images or references, the notation P17 corresponds to 1st_T_P1 from the original naming scheme,
P18 corresponds to 1st_T_P2,
P19 corresponds to 1st_T_P3,
P20 corresponds to 1st_T_P4.

5. sheetmusic/

• Contents: Sheet music used in the recording sessions.
• Details:
  • S1 folder has warm-up materials (e.g., Pachelbel’s Canon, major scales).
  • S2 folder includes 18 classical pieces arranged for flute-piano duet, each assigned a specific tempo (50, 100, 150 BPM).

S1	Scale/Canon		S2	18 Pieces
No.	Composer	Tempo	No.	Composer	Tempo
1	Scale	50	1	Gluck	50
2	Canon	100	2	Bach	100
3	Scale	150	3	Donjon	150
4	Canon	50	4	Chedeville	50
5	Scale	100	5	Mozart	100
6	Canon	150	6	Popp	150
7	Canon	50	7	Popp	150
8	Canon	100	8	Bach	100
9	Scale	150	9	Donjon	150
10	Canon	50	10	Chedeville	50
11	Scale	100	11	Mozart	100
12	Canon	150	12	Popp	150
			13	Popp	150
			14	Bach	100
			15	Donjon	150
			16	Chedeville	50
			17	Mozart	100
			18	Popp	150

6. wav/

• Contents: .wav audio recordings for each flutist (P1–P20).
• Details:
  • Possibly arranged in subfolders named by flutist or trial index.
  • Recordings capture simultaneous flute–piano duets at various tempos/pieces.

Usage Notes

• Synchronization:
  All recordings and annotations are aligned via an experimenter’s clap (Start). The annotation CSV has timestamps for all relevant events.

• Gesture/Breath Cues:

  • In motion_numpy/, each .npy contains summarized peak frames (pos, vel, acc) and event frames for easy reference.
  • In motion_pkl/, you can find more detailed or raw optical-flow data, allowing custom processing.

• Sheet Music & Performance Setup:

  • The sheetmusic/ folder shows the exact materials used by the flutists and pianists.
  • Tempo was set at 50, 100, or 150 BPM for each piece, repeated multiple times.

Reference

If you use this dataset, please cite:

Predicting Flutist Onset Timing in Duet Performance: A Multimodal Analysis of Gesture and Breath Cues
(Anonymous Authors, 2025)

For the methodology, analysis, and initial results, see the paper linked in this repository. If you have questions, feel free to open an issue or contact the authors.

This project includes the main components:

• Cue dataset (MIT License, CC BY 4.0)

License

• The code is licensed under the MIT License. See LICENSE.
• The dataset is licensed under CC BY 4.0. See LICENSE_DATA.