[Feature request] Return timestamps for TTS output

[AlexLavSi]

Hey, would you mind adding some sweet new features? It would be amazing if we could get timelines for sentences in the text, maybe as a JSON file or something.

[eginhard]

It's not something we're planning to do ourselves in the near term, but I'd merge PRs adding this. If someone wants to work on it, best to submit a rough plan for feedback here first to agree on a common structure that would work with all Coqui models.

[AlexLavSi]

Thank you for your response.
I hope that this improvement will be both needed and liked by someone and that it will be made quickly. I would like to propose this variant of the JSON file (based on my needs):

{ "duration": 500, "timestamps": [ { "text": "bla bla bla 1.", "start_time": 0.0, "end_time": 3.5 }, ... { "text": "bla bla bla end.", "start_time": 480.7, "end_time": 500 } ] }

And text parsing can be done by sentences in the main text.

[santosh-r24]

Hello, @eginhard
Thanks for taking the effort to maintaining this project!

I wanted to see if I can take a crack at this.
Here's an outlined plan for a model agnostic implementation for sentence level timestamps.

Plan
The approach is to implement this at the Synthesizer.tts level.

1. Add use_timestamps flag, to make it backward compatible
2. If the flag is true, compute the duration of waveform by doing
   wav_duration = len(waveform) / self.tts_config.audio["sample_rate"].
   The structure of the timestamps itself could be something like
   {"text": sen, "start": current_time, "end": current_time + duration_sec}
   Then, update current_time for the next loop including the padding.
3. if use_timestamps=False, the function returns wavs as it does now.
   Else, it'd return a dictionary {"wav": wavs, "timestamps": timestamp_list}
4. In api.py, tts() and tts_to_file() methods would also be updated to accept a use_timestamps flag and handle the new dictionary return type.

Dependencies

1. The implementation is contingent on split_sentences=True, to get sentence-level timestamps. If set to False, it'd return a single timestamp for the entire block of text.
2. The sentence segmenter is currently hardcoded to "en" related to issue Setting the synthesizer sentence segmenter for a given language #324 . The current logic would correctly timestamp sentence chunks provided by segmenter, so this is a known dependancy, but not a blocker as such.

Please let me know if this aligns with your vision, and if there's any feedback on the structure itself.

If this sounds good, I'd be happy to get started with the PR.
Thanks !

[eginhard]

@santosh-r24 Thanks for your interest in this. Feel free to make a start and open a draft PR. Some suggestions:

• Eventually returning word-level timestamps will be useful as well for models that support it. The output format should already take this into consideration to avoid conflicts later. We could follow this library: https://github.com/linto-ai/whisper-timestamped?tab=readme-ov-file#example-output
• Use a return_dict flag instead of use_timestamps to support other additional outputs in the future
• For now, only update the Synthesizer class, not the API

[santosh-r24]

Thanks @eginhard! That sounds like a solid plan.

I'll proceed with the return_dict flag and the Whisper-style output structure (wav, outputs, segments). I'll keep the changes strictly to Synthesizer.tts for this PR.

Regarding word-level timestamps: I'd be interested in picking that up as a follow-up task for models that support it, down the pipeline 😄.

I'll open a Draft PR for this shortly to track progress.