Skip to content

Commit c330c7a

Browse files
auyidi1auyidi
auyidi1
and
auyidi
authored
feat(skills): add tts-voiceover skill for Azure Speech SDK voice-over generation (#1415)
# Pull Request ## Description Add a new `tts-voiceover` skill to the experimental collection. This skill generates per-slide WAV voice-over files from YAML speaker notes using the Azure Speech SDK with SSML pronunciation control for technical acronyms. The skill integrates with the existing PowerPoint skill's `content.yaml` speaker notes and supports both key-based and Microsoft Entra ID token authentication. ### Key features - `generate_voiceover.py`: Reads `content.yaml` files, applies SSML acronym aliases, generates WAV files per slide - `embed_audio.py`: Embeds WAV files into PPTX decks via python-pptx - Configurable voice (`--voice`), speech rate (`--rate`), and custom acronym lexicon (`--lexicon`) - Dry-run mode for SSML template verification without Azure credentials - Supports both `SPEECH_KEY` (key auth) and `SPEECH_RESOURCE_ID` (Entra ID auth) ## Related Issue(s) Fixes #1417 ## Type of Change **Code & Documentation:** * [x] New feature (non-breaking change adding functionality) * [x] Documentation update **Infrastructure & Configuration:** * [x] GitHub Actions workflow **AI Artifacts:** * [x] Copilot skill (`.github/skills/*/SKILL.md`) **Other:** * [x] Script/automation (`.ps1`, `.sh`, `.py`) ## Sample Prompts (for AI Artifact Contributions) **User Request:** > Generate voice-over narration for the slide deck speaker notes **Execution Flow:** 1. Load the tts-voiceover skill 2. Run `python scripts/generate_voiceover.py --dry-run --content-dir content` to verify SSML 3. Run `python scripts/generate_voiceover.py --content-dir content --output-dir voice-over` to generate WAV files 4. Run `python scripts/embed_audio.py --input deck.pptx --audio-dir voice-over` to embed audio **Output Artifacts:** - `voice-over/slide-001.wav` through `slide-NNN.wav` (one WAV per slide) - `*-narrated.pptx` (PPTX with embedded audio) **Success Indicators:** - All slides with `speaker_notes:` produce WAV files - Acronyms (OWASP, SBOM, SLSA, etc.) are pronounced correctly via SSML aliases - `--dry-run` output shows valid SSML with `<sub alias>` elements ## Testing - `npm run validate:skills` passes (17/17 skills, 0 errors) - `npm run lint:py` passes (ruff clean) - `npm run lint:frontmatter` passes (0 errors) - `npm run lint:collections-metadata` passes (0 errors, 14 collections) - `npm run plugin:generate` completes without errors - `npm run spell-check` passes for changed files (SSML added to cspell dictionary) - Fuzz harness covers `apply_acronym_aliases`, `wrap_ssml`, `load_acronyms` - `test_embed_audio.py`: 6 tests covering WAV duration, narration timing XML, embed success/failure - `test_generate_voiceover.py`: 9 tests covering lexicon resolution, parser defaults, dry-run paths - 28 total tests pass ## Checklist ### Required Checks - [x] Documentation is updated (if applicable) - [x] Files follow existing naming conventions - [x] Changes are backwards compatible (if applicable) - [x] Tests added for new functionality (if applicable) ### AI Artifact Contributions - [ ] Used `/prompt-analyze` to review contribution — N/A: skill contains Python scripts, not prompt/agent artifacts - [ ] Addressed all feedback from `prompt-builder` review — N/A: see above - [x] Verified contribution follows common standards and type-specific requirements ### Required Automated Checks - [x] Markdown linting: `npm run lint:md` - [x] Spell checking: `npm run spell-check` - [x] Frontmatter validation: `npm run lint:frontmatter` - [x] Skill structure validation: `npm run validate:skills` - [x] Link validation: `npm run lint:md-links` - [x] PowerShell analysis: `npm run lint:ps` - [x] Plugin freshness: `npm run plugin:generate` - [x] Docusaurus tests: `npm run docs:test` ## Security Considerations - [x] This PR does not contain any sensitive or NDA information - [x] Any new dependencies have been reviewed for security issues - [x] Security-related scripts follow the principle of least privilege ## Additional Notes - Fixed SSML XML injection: speaker notes are now XML-escaped before acronym alias substitution - Added Entra ID token refresh logic: tokens refresh 5 min before expiry for long-running decks - Removed unrelated `mcp-security.instructions.md` reference from `hve-core-all.collection.yml` - Added SSML to cspell dictionary - DragonHD voices (en-US-Andrew, en-US-Davis, en-US-Ava) produce the most natural results --------- Co-authored-by: auyidi <auyidi@microsoft.com>
1 parent ba520aa commit c330c7a

29 files changed

Lines changed: 3338 additions & 0 deletions

‎.cspell/general-technical.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1595,3 +1595,4 @@ LASTEXITCODE
15951595
scriptblock
15961596
DSSE
15971597
intoto
1598+
SSML

‎.github/skills/experimental/tts-voiceover/SKILL.md‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
---
2+
name: tts-voiceover
3+
description: 'Text-to-speech voice-over generation from YAML speaker notes using Azure Speech SDK with SSML pronunciation control - Brought to you by microsoft/hve-core'
4+
metadata:
5+
authors: "microsoft/hve-core"
6+
spec_version: "1.0"
7+
---
8+
9+
# TTS Voice Over Skill
10+
11+
Generates per-slide WAV voice-over files from YAML `speaker_notes` using Azure Speech SDK with SSML pronunciation control.
12+
13+
## Overview
14+
15+
This skill reads `content.yaml` files from a PowerPoint skill content directory, extracts `speaker_notes` fields, applies SSML acronym aliases for correct pronunciation of technical terms, and produces one WAV file per slide. Supports dry-run mode for SSML template verification without Azure credentials.
16+
17+
## Prerequisites
18+
19+
* **Azure Speech resource** — Free tier provides 500K characters per month.
20+
* **Authentication** — Key-based (`SPEECH_KEY`) or Microsoft Entra ID (`SPEECH_RESOURCE_ID`).
21+
* **Python 3.11+** with `uv` for virtual environment management.
22+
23+
### Key-Based Auth
24+
25+
```bash
26+
export SPEECH_KEY="your-speech-key"
27+
export SPEECH_REGION="eastus"
28+
```
29+
30+
### Microsoft Entra ID Auth
31+
32+
Requires a custom domain on the Speech resource and `Cognitive Services Speech User` role.
33+
34+
```bash
35+
export SPEECH_RESOURCE_ID="/subscriptions/.../Microsoft.CognitiveServices/accounts/your-resource"
36+
export SPEECH_REGION="eastus"
37+
```
38+
39+
Install dependencies:
40+
41+
```bash
42+
# run from this skill folder
43+
uv sync
44+
```
45+
46+
## Quick Start
47+
48+
Verify SSML templates without generating audio:
49+
50+
```bash
51+
uv run scripts/generate_voiceover.py --dry-run --content-dir path/to/content
52+
```
53+
54+
Generate voice-over WAV files:
55+
56+
```bash
57+
uv run scripts/generate_voiceover.py --content-dir path/to/content --output-dir voice-over
58+
```
59+
60+
Embed audio into a PPTX deck:
61+
62+
```bash
63+
uv run scripts/embed_audio.py --input deck.pptx --audio-dir voice-over --output deck-narrated.pptx
64+
```
65+
66+
## Parameters Reference
67+
68+
### generate_voiceover.py
69+
70+
| Parameter | Type | Default | Description |
71+
|:----------------|:-------|:------------------------------------|:----------------------------------------------|
72+
| `--dry-run` | flag | `false` | Print SSML templates without generating audio |
73+
| `--voice` | string | `en-US-Andrew:DragonHDLatestNeural` | Azure TTS voice name |
74+
| `--rate` | string | `+10%` | Speech prosody rate |
75+
| `--content-dir` | path | `content` | Path to slide content directory |
76+
| `--output-dir` | path | `voice-over` | Path to WAV output directory |
77+
| `--lexicon` | path | *(auto-detect)* | Custom acronyms.yaml path |
78+
| `--verbose` / `-v` | flag | `false` | Enable verbose (DEBUG) logging output |
79+
80+
### embed_audio.py
81+
82+
Embeds WAV files into corresponding PPTX slides and adds narration timing
83+
XML so PowerPoint recognizes the audio for video export via
84+
**File > Export > Create a Video > Use Recorded Timings and Narrations**.
85+
86+
| Parameter | Type | Default | Description |
87+
|:--------------|:-----|:------------------|:-----------------------------|
88+
| `--input` | path | *(required)* | Source PPTX file path |
89+
| `--audio-dir` | path | `voice-over` | Directory with slide-NNN.wav |
90+
| `--output` | path | `*-narrated.pptx` | Output PPTX file path |
91+
| `--verbose` / `-v` | flag | `false` | Enable verbose (DEBUG) logging output |
92+
93+
## Script Reference
94+
95+
Generate with custom voice and rate:
96+
97+
```bash
98+
uv run scripts/generate_voiceover.py \
99+
--content-dir content \
100+
--output-dir voice-over \
101+
--voice "en-US-Jenny:DragonHDLatestNeural" \
102+
--rate "+5%"
103+
```
104+
105+
Use a custom lexicon:
106+
107+
```bash
108+
uv run scripts/generate_voiceover.py \
109+
--content-dir content \
110+
--lexicon custom-acronyms.yaml
111+
```
112+
113+
Embed generated audio:
114+
115+
```bash
116+
uv run scripts/embed_audio.py \
117+
--input slide-deck/presentation.pptx \
118+
--audio-dir voice-over \
119+
--output slide-deck/presentation-narrated.pptx
120+
```
121+
122+
## Acronym Lexicon
123+
124+
The lexicon controls SSML `<sub alias>` replacements for acronyms and technical terms. Create an `acronyms.yaml` file:
125+
126+
```yaml
127+
acronyms:
128+
HVE-Core: "H V E Core"
129+
OWASP: "Oh wasp"
130+
SBOM: "S Bomb"
131+
SLSA: "Salsa"
132+
CI/CD: "C I C D"
133+
```
134+
135+
Lexicon resolution order:
136+
137+
1. Path specified via `--lexicon` argument.
138+
2. `acronyms.yaml` in the content directory.
139+
3. Built-in defaults covering common technical acronyms.
140+
141+
## SSML Template
142+
143+
Each slide produces an SSML document:
144+
145+
```xml
146+
<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis"
147+
xmlns:mstts="http://www.w3.org/2001/mstts" xml:lang="en-US">
148+
<voice name="en-US-Andrew:DragonHDLatestNeural">
149+
<prosody rate="+10%">
150+
Text with <sub alias="Oh wasp">OWASP</sub> aliases applied.
151+
</prosody>
152+
</voice>
153+
</speak>
154+
```
155+
156+
## Integration with PowerPoint Skill
157+
158+
This skill reads from the PowerPoint skill's content directory structure:
159+
160+
```text
161+
content/
162+
├── slide-001/
163+
│ └── content.yaml # Must include speaker_notes: field
164+
├── slide-002/
165+
│ └── content.yaml
166+
└── ...
167+
```
168+
169+
Each `content.yaml` should contain a `speaker_notes:` field with the narration text. The generated WAV files are named `slide-NNN.wav` matching the directory names.
170+
171+
## Troubleshooting
172+
173+
| Issue | Solution |
174+
|:-----------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------|
175+
| `Set SPEECH_KEY ... or SPEECH_RESOURCE_ID` | Export `SPEECH_KEY` (key auth) or `SPEECH_RESOURCE_ID` (Entra ID) with `SPEECH_REGION`. |
176+
| 401 with Entra ID auth | Verify custom domain on the Speech resource and `Cognitive Services Speech User` role. RBAC propagation takes up to 5 minutes. |
177+
| Empty WAV files or skipped slides | Verify `speaker_notes:` is present and non-empty in `content.yaml`. |
178+
| Mispronounced acronyms | Add entries to `acronyms.yaml` with phonetic aliases. |
179+
| `azure-cognitiveservices-speech package is required` | Run `uv sync` in the skill directory. |
180+
| Audio icon visible in PPTX | Reposition or resize the audio object in PowerPoint after embedding. |
181+
| Authored slide animations missing after embedding | `embed_audio.py` replaces existing `p:timing` with narration timing; re-apply animations in PowerPoint after embedding audio. |
182+
| Slides no longer advance on click after embedding | `embed_audio.py` sets `advClick="0"` for auto-advance. To re-enable, select all slides in PowerPoint and check **Advance Slide > On Mouse Click** in the Transitions tab. |
183+
| Video export shows "No timings recorded" | Re-embed audio with the updated `embed_audio.py` which adds narration timing XML automatically. |
184+
185+
> Brought to you by microsoft/hve-core
186+
187+
🤖 Crafted with precision by ✨Copilot following brilliant human instruction, then carefully refined by our team of discerning human reviewers.

‎.github/skills/experimental/tts-voiceover/pyproject.toml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
[project]
2+
name = "tts-voiceover-skill"
3+
version = "0.0.0"
4+
requires-python = ">=3.11"
5+
dependencies = [
6+
"azure-cognitiveservices-speech>=1.41",
7+
"azure-identity>=1.19",
8+
"lxml>=6.1.0", # direct dep (embed_audio.py) and transitive via python-pptx; explicit pin ensures CVE patches
9+
"python-pptx>=1.0",
10+
"pyyaml>=6.0",
11+
]
12+
13+
[dependency-groups]
14+
dev = [
15+
"pytest>=9.0",
16+
"pytest-cov>=5.0",
17+
"pytest-mock>=3.14",
18+
"ruff>=0.15",
19+
]
20+
fuzz = [
21+
"atheris>=3.0",
22+
]
23+
24+
[tool.pytest.ini_options]
25+
testpaths = ["tests"]
26+
pythonpath = ["scripts"]
27+
python_files = ["test_*.py", "fuzz_harness.py"]
28+
29+
[tool.ruff]
30+
line-length = 88
31+
target-version = "py311"
32+
33+
[tool.ruff.lint]
34+
select = ["E", "F", "I", "W"]

‎.github/skills/experimental/tts-voiceover/scripts/Invoke-EmbedAudio.ps1‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
#!/usr/bin/env pwsh
2+
# Copyright (c) Microsoft Corporation.
3+
# SPDX-License-Identifier: MIT
4+
#Requires -Version 7.0
5+
#
6+
# Invoke-EmbedAudio.ps1
7+
#
8+
# Purpose: Wrapper that manages uv venv setup and delegates to embed_audio.py
9+
10+
<#
11+
.SYNOPSIS
12+
Embeds per-slide WAV voice-over files into a PowerPoint deck.
13+
14+
.DESCRIPTION
15+
Manages the Python virtual environment and invokes embed_audio.py to add
16+
WAV files as embedded media objects in the corresponding slides of a PPTX file.
17+
18+
.PARAMETER InputPath
19+
Source PPTX file path. Required.
20+
21+
.PARAMETER AudioDir
22+
Directory containing slide-NNN.wav files. Defaults to voice-over.
23+
24+
.PARAMETER OutputPath
25+
Output PPTX file path. Defaults to input stem + '-narrated.pptx'.
26+
27+
.PARAMETER SkipVenvSetup
28+
Skip virtual environment creation and dependency installation.
29+
30+
.EXAMPLE
31+
./Invoke-EmbedAudio.ps1 -InputPath deck.pptx -AudioDir voice-over
32+
33+
.EXAMPLE
34+
./Invoke-EmbedAudio.ps1 -InputPath deck.pptx -AudioDir voice-over -OutputPath deck-narrated.pptx
35+
36+
.NOTES
37+
Part of the tts-voiceover skill. Manages uv virtual environment setup
38+
and delegates to embed_audio.py for WAV embedding into PPTX slides.
39+
#>
40+
41+
[CmdletBinding()]
42+
param(
43+
[Parameter(Mandatory = $true)]
44+
[ValidateNotNullOrEmpty()]
45+
[string]$InputPath,
46+
47+
[Parameter(Mandatory = $false)]
48+
[string]$AudioDir,
49+
50+
[Parameter(Mandatory = $false)]
51+
[string]$OutputPath,
52+
53+
[Parameter(Mandatory = $false)]
54+
[switch]$SkipVenvSetup
55+
)
56+
57+
$ErrorActionPreference = 'Stop'
58+
59+
$ScriptDir = $PSScriptRoot
60+
$SkillRoot = Split-Path $ScriptDir
61+
$VenvDir = Join-Path $SkillRoot '.venv'
62+
63+
Import-Module (Join-Path $ScriptDir 'Modules/TtsVoiceoverHelpers.psm1') -Force
64+
65+
#region Main
66+
67+
if ($MyInvocation.InvocationName -ne '.') {
68+
69+
$null = Test-UvAvailability
70+
71+
if (-not $SkipVenvSetup) {
72+
Initialize-PythonEnvironment -SkillRoot $SkillRoot
73+
}
74+
75+
$python = Get-VenvPythonPath -VenvDir $VenvDir
76+
if (-not (Test-Path $python)) {
77+
throw "Python not found at $python. Run without -SkipVenvSetup to initialize."
78+
}
79+
80+
$script = Join-Path $ScriptDir 'embed_audio.py'
81+
$PythonArgs = @('--input', $InputPath)
82+
83+
if ($AudioDir) { $PythonArgs += '--audio-dir', $AudioDir }
84+
if ($OutputPath) { $PythonArgs += '--output', $OutputPath }
85+
if ($VerbosePreference -ne 'SilentlyContinue') { $PythonArgs += '--verbose' }
86+
87+
& $python $script @PythonArgs
88+
if ($LASTEXITCODE -ne 0) {
89+
throw "embed_audio.py exited with code $LASTEXITCODE"
90+
}
91+
92+
}
93+
94+
#endregion Main

0 commit comments

Comments
 (0)