Warning
Perso AI is introducing a new SDK — Perso Interactive On-Device SDK, now officially available under a new GitHub organization and repository.
This repository has been archived and will no longer receive updates. Please migrate to the new SDK for the latest features, improvements, and support.
- New Repository: https://github.com/perso-ai/perso-interactive-ondevice-sdk-swift
- Documentation: https://perso-ai.github.io/perso-interactive-ondevice-sdk-swift/
The Perso AI LiveChat SDK is the next-generation universal interface for conversational AI. It supports real-time communication and personalized interactions with over 100 languages, featuring natural speech-to-text conversion and lifelike AI Human expressions.
The Perso AI LiveChat SDK supports :
- macOS 15.0 or later
- Xcode 16.0 or later
- Swift 6.0 or later
To install the Perso AI LiveChat SDK, follow these steps:
- Open your Xcode project.
- Navigate to
File>Add Package Dependencies.... - Enter the package repository URL:
https://github.com/est-perso-live/perso-livechat-ondevice-sdk-swift. - Choose the version range or specific version.
- Click
Add Packageto add PersoLiveChatOnDeviceSDK to your project
Once this setup is complete, you can import PersoLiveChatOnDeviceSDK and start using the SDK in your Swift code.
The SDK includes features that may require microphone access. However, microphone permission is optional and will only be requested if microphone-dependent features are used (e.g., voice chat).
To ensure proper functionality for these features, include the following permission key in your app's Info.plist file:
| Key | Value (Example) |
|---|---|
Privacy - Microphone Usage Description |
This app requires microphone access for voice chat features. |
Import the SDK where you need to use it.
import PersoLiveChatOnDeviceSDKIf you choose to use an API Key, you must set it with top priority before any SDK feature is used.
PersoLiveChat.apiKey = "YOUR_API_KEY"Caution
The Perso AI LiveChat SDK for Swift is not recommended to embed an API Key directly in your app. There's a high risk that malicious actors could discover and misuse your API Key. We strongly recommend using a SessionID issued by your backend whenever possible, rather than exposing your API Key.
When you have a server-issued SessionID, you do not need to configure an API Key. Your app only needs the assigned SessionID to establish a connection with the SDK, greatly reducing any risk of API Key exposure.
Skip API Key setup.
To use PersoLiveChat, AI human model information must be registered in the system.
The process of retrieving the registered model style information is required first.
The list of available model styles can be retrieved using the following method:
let styles = try await PersoLiveChat.fetchAvailableModelStyles()This method does more than simply return a list of styles.
It also provides metadata for each style, including the availability and status of its associated model resources.
This allows the app to make informed decisions about whether a style is ready for use or needs to be downloaded.
Asynchronously downloads and prepares all model resources required for the specified ModelStyle.
Resources are downloaded concurrently, and progress updates are emitted through AsyncThrowingStream<Progress, Error>,
allowing you to observe download progress in real time and handle errors gracefully:
do {
let stream = PersoLiveChat.loadModelStyle(with: ModelStyle)
for try await progress in stream {
// handle progress
}
} catch {
// handle error
}Downloads are performed only if there are changes or updates to the resources.
If the server responds with .notModified, the existing cached version is used instead of re-downloading.
This reduces unnecessary network usage and improves overall performance.
Removes files and resets internal tracking information related to model styles.
PersoLiveChat.cleanModelResources()Before using any PersoLiveChat features, you must load and initialize the models required for inference with a specific model style.
This process requires model resources, which must be downloaded beforehand to ensure proper initialization.
If the initialization method is not called, errors may occur during prediction.
try await PersoLiveChat.load(ModelStyle)This project offers a model warmup feature to optimize initial prediction performance. By using this feature, the time required for predictions during the first call of PersoLive is reduced, allowing users to experience faster response times.
Key Features:
- Model Warmup: Loads and initializes the model at the start of the application, minimizing potential delays when users make their first prediction request.
- Reduced Initial Prediction Time: Utilizes the pre-warmed model to handle the first prediction request in PersoLive more swiftly. By using this feature, you can enhance the application's responsiveness and improve the overall user experience.
It is recommended to call this at an early stage, before starting PersoVideoView.
await PersoLiveChat.warmup()Note:
AVAudioSessionis only available oniOSandvisionOS. FormacOS, no audio session setup is required.
PersoLiveChat uses AVAudioSession to perform voice-related functions. It is essential to set up AVAudioSession before starting PersoVideoView.
func setAudioSession(
category: AVAudioSession.Category,
mode: AVAudioSession.Mode = .default,
policy: AVAudioSession.RouteSharingPolicy = .default,
options: AVAudioSession.CategoryOptions = []
) throwsIf the AVAudioSession fails to be configured or activated, the PersoLiveChatError.audioSessionSetupError will be thrown. Make sure to handle errors properly to prevent unexpected crashes or issues.
Example:
do {
try PersoLiveChat.setAudioSession(
category: .playAndRecord,
options: [.defaultToSpeaker, .allowBluetooth, .allowBluetoothA2DP]
)
} catch {
print("Failed to set up audio session: \(error)")
}
// After setting the audio session, start PersoVideoView.The appropriate
AVAudioSession.CategoryandAVAudioSession.CategoryOptionsshould be configured based on your specific use case.For more detailed information about categories, options, and other related topics, please refer to the AVAudioSession SetCategory documentation
Before creating a session, it is necessary to define which features will be used to create the session. It is essential to first obtain the model information available for each feature in STT, LLM, and TTS.
// Obtain usable STT Model information.
try await PersoLiveChat.fetchAvailableSTTModels()
// Obtain usable prompts information.
try await PersoLiveChat.fetchAvailablePrompts()
...
...
...Not everything can be demonstrated in the example, so please refer to the specifications for the required features.
When using a SessionID, it is not necessary to use it.
You must first be assigned a SessionID before using the features. Add the feature you want to use and include the necessary configuration information.
do {
session = try await PersoLiveChat.createSession(
for: [
.speechToText(type: STTType),
.largeLanguageModel(
llmType: LLMType,
promptID: String,
documentID: String?
),
.textToSpeech(type: TTSType)
],
modelStyle: ModelStyle
) { status in
// session status handler
}
} catch {
// handle session create error
}It can be used if a SessionID has already been assigned.
Within the SDK, the validity of the SessionID is checked, and if it is valid, a session can be assigned.
do {
session = try await PersoLiveChat.connectSession(
sessionID: "assigned SessionID",
modelStyle: ModelStyle
) { status in
// session status handler
}
} catch {
// handle session create error
}When you're done with the session and want to end it, you can use the stopSession() method:
PersoLiveChat.stopSession()This method ends the current session if one exists. It's useful when the user closes the livechat interface. After calling this method, the current session object becomes invalid, and you'll need to create a new session or connect to an existing one to start a new chat.
You can also listen to the session status handler to respond to session termination events:
session = try await PersoLiveChat.createSession(
for: [
.speechToText(type: STTType)
],
modelStyle: ModelStyle
) { status in
switch status {
case .started:
print("Start Session")
case .terminated:
print("Session has been terminated.")
// Update UI or take other actions
}
}You can control how the video content is displayed on the screen by configuring the videoContentMode and offsetY properties of PersoVideoView.
The videoContentMode property allows you to choose between aspectFill and aspectFit modes, while the offsetY property enables vertical positioning adjustment of the AI Human within the view.
let persoVideoView = PersoVideoView(session: session)
// Set the video content mode (default is aspectFill)
persoVideoView.videoContentMode = .aspectFit
// Adjust vertical offset position (default is 0)
persoVideoView.offsetY = 10.0
persoVideoView.start()If you want to see more information, please navigate to PersoVideoView.
STT provides a feature that converts recognized speech from voice data into text. The converted text can be used to compose a chat screen or tailored to meet the user's needs.
do {
// input recording audio data or voice data
let userText = try await session.transcribeAudio(audio: audio, language: "ko")
// using converted text
} catch {
// handle STT error
}Completes the chat by generating a response based on the input message using a Large Language Model (LLM).
do {
let sentenceStream = session.completeChat(message: message)
var contents: [String] = []
for try await sentence in sentenceStream {
// delivered sentence by sentence
}
} catch {
// handle LLM Completes the chat error
}TTS provides speech synthesis services from text. You only need to send the text you want to synthesize into speech to PersoVideoView.
// Results obtained through LLM Chat Completion.
persoVideoView.push(text: String)It is possible to implement the desired features yourself while using other features provided by the SDK. If you want to learn about customization, please navigate to the Customization.
Providing a feature to create and display AI Human capable of real-time conversation based on text generated through the pipeline.
PersoVideoView is required PersoLiveChatSession.
To use PersoVideoView, the start() method must be called, but there's an important point to consider here.
This method must be called while the app is in the foreground. Attempting to start while the app is in the background will result in a failure to initialize the audio session properly and can cause unexpected errors. If this method is called while the app is in the background, an error will be emitted asynchronously through the delegate method PersoLiveChatDelegate.didFailWithError(_:), instead of immediately throwing an error.
The PersoVideoViewDelegate protocol defines a set of methods that can be implemented by a delegate to respond to events that occur in a PersoVideoView. By implementing this protocol, you can capture and handle events that arise during the operation of the PersoVideoView in your application.
Specifically, the persoVideoView(didChangeState state: PersoVideoView.VideoState) method detects changes in the state of the PersoVideoView, indicating whether it is in the waiting state (waiting for a question) or in the processing state (responding). This can be used to handle the UI or implement necessary business logic accordingly.
public enum VideoState {
// The view is idle and waiting for new input or a new action.
case waiting(_ phase: Phase)
// The view is actively processing speech synthesis or video-related tasks.
case processing
public enum Phase {
case idle
case transition
case standby
}
}
public protocol PersoVideoViewDelegate: AnyObject {
func persoVideoView(didFailWithError error: PersoLiveChatError)
func persoVideoView(didChangeState state: PersoVideoView.VideoState)
}
// usage
let videoView = PersoVideoView(session: PersoLiveChatSession)
videoView.delegate = selfNote: - These methods are
optional, and they do not need to be implemented if they are unnecessary.
To manage the visibility of the AI Human, use the following methods:
/// Start displaying the AI Human
persoVideoView.start()
/// Stop the AI Human display
persoVideoView.stop()
/// Pause the AI Human and Sounds.
persoVideoView.pause()These methods allow you to control when the AI Human is visible and active on the screen.
When creating a session, an intro message can be specified in the prompt information. If an intro message is set, it can be played immediately at the start using the playIntro().
The executed intro message can be received through the closure, and additional actions can be performed using the received information.
/// An intro message is needed in the prompt.
persoVideoView.playIntro { message in
// handle message such as UI.
}To make the AI human speech, simply pass the text you want to synthesize into the PersoVideoView using the push(text:) method.
// Start the video view
...
// Make the AI human speech the given text
persoVideoView.push(text: "Hello, how can I assist you today?")Make sure to handle session status changes, as the session must be active when sending text to the AI for speech synthesis.
If you want to cancel an AI Human response immediately because it's not desired or for any other reason, you can use the stopSpeech.
// Stops the speech synthesis immediately and clears the audio buffer.
await persoVideoView.stopSpeech()STT, TTS, and LLM can be customized and implemented according to your needs. There are a few rules.
A custom class that conforms to the SpeechSynthesizable protocol needs to be implemented.
The following information pertains to the SpeechSynthesizable protocol interface.
public protocol SpeechSynthesizable {
/// Synthesizes speech from the given text (Text-to-Speech).
///
/// - Parameters:
/// - text: The text string that should be synthesized into speech.
/// - Returns: A `Data` object representing the synthesized speech audio.
/// - Throws: An error if the synthesis process fails.
func synthesizeSpeech(text: String) async throws -> Data
}If you look at the createSession or connectSession initializer information, you can see that the provider can be injected.
public func createSession(
for config: Set<SessionCapability> = [],
modelStyle: ModelStyle,
provider: (any SpeechSynthesizable)? = nil,
statusHandler: @escaping (PersoLiveChatSession.SessionStatus) -> Void
) async throws -> PersoLiveChatSession {
...
}
public func connectSession(
sessionID: String,
modelStyle: ModelStyle,
provider: (any SpeechSynthesizable)? = nil,
statusHandler: @escaping (PersoLiveChatSession.SessionStatus) -> Void
) async throws -> PersoLiveChatSession {
...
}PersoMLComputeUnits property specifies the compute resources to be used when executing a Core ML model. By selecting the appropriate compute units, you can balance performance optimization and power efficiency.
The default value is set to .ane.
public enum PersoMLComputeUnits {
case ane
case cpu
}
PersoLiveChat.computeUnits = .cpu.aneis available on devices with Apple Neural Engine (A13 Bionic and later) and is recommended for best performance..cpuis universally supported and is ideal for testing or as a fallback strategy.
Perso AI LiveChat SDK for Swift is commercial software. Contact our sales team.