Key Interfaces

1. Initialization

Initialize the module with the appId and appSecret provided in your platform account and get the Play object

VirtualFactory.init("your appId", "your appSecret")
player = VirtualFactory.getPlayer(mContext, eglBaseContext)

2. Connect to Digital Human

Connect to the digital human using the conversationId provided by your platform account

    player?.addCallback(callback)
    player?.connect("your conversation id")

The callback contains the following callbacks

`onShow` Digital human is ready to display

void onShow();

`onError` Exception occurred when connecting to digital human

void onError(int msgType, int msgSubType, String msg);

Parameter description:

Parameter Name	Type	Description
msgType	int	Error type
msgSubType	int	Sub-error type
msg	String	Exception message

Values of msgType:

Value	Description
1000	Authorization exception
1001	Session creation exception
1002	Resource retrieval exception
1010	IM connection creation failed
1011	Rendering service returned exception
1020	RTC status exception
1030	Rendering service actively closed
1040	IM connection lost
1050	RTC connection lost

`onVideoTrack` RTC media channel created successfully

Bind the digital human widget to the track in this callback

void onVideoTrack(VideoTrack track);

Parameter Name	Type	Description
track	VideoTrack	Video media channel

`onAudioSamples` Local audio sampling data callback

You can implement audio visualization and other operations in this callback

default void onAudioSamples(int audioFormat, int channelCount, int sampleRat, byte[] data){}

`onTtsSpeakStart` Digital human starts playing TTS synthesized audio

default void onTtsSpeakStart(){}

`onTtsSpeakText` Text content of TTS played by digital human

default void onTtsSpeakText(String text){}

`onTtsSpeakStop` TTS text playback completed

default void onTtsSpeakStop(){}

`onSpeakStart` Digital human starts speaking

default void onSpeakStart(){}

`onSpeakText` Text content spoken by digital human

default void onSpeakText(String text){}

`onSpeakStop` Digital human finished speaking

default void onSpeakStop(){}

`onAsrResult` ASR recognition content callback

default void onAsrResult(String text, boolean sentenceEnd){}

3. Drive Digital Human to Speak with Audio URL

Use a WAV audio URL with 16kHz sampling rate, 16bit, mono channel to drive the digital human to speak

player?.speakWithWav(wavUrl, true)

Parameter Name	Type	Description
wavUrl	String	Network address of WAV audio
interrupt	boolean	Whether to interrupt the current speaking state

4. Drive Digital Human to Speak with Text

Input the text that you want the digital human to say, and the digital human will say the corresponding content according to the voice configured in the session.

player?.speakWithTxt(text, true)

Parameter Name	Type	Description
text	String	Text content for the digital human to say
interrupt	boolean	Whether to interrupt the current speaking state

5. Digital Human Q&A

Let the digital human answer your questions.

player?.speakWithQuestion(text, true)

Parameter Name	Type	Description
text	String	Express your question to the digital human
interrupt	boolean	Whether to interrupt the current speaking state

6. Interrupt Digital Human Speaking

Make the digital human stop talking

player?.stopAudio()

API Documentation

Replica

Conversational Video Interface

iOS SDK

Android SDK

H5 SDK

1. Initialization

2. Connect to Digital Human

`onShow` Digital human is ready to display

`onError` Exception occurred when connecting to digital human

`onVideoTrack` RTC media channel created successfully

`onAudioSamples` Local audio sampling data callback

`onTtsSpeakStart` Digital human starts playing TTS synthesized audio

`onTtsSpeakText` Text content of TTS played by digital human

`onTtsSpeakStop` TTS text playback completed

`onSpeakStart` Digital human starts speaking

`onSpeakText` Text content spoken by digital human

`onSpeakStop` Digital human finished speaking

`onAsrResult` ASR recognition content callback

3. Drive Digital Human to Speak with Audio URL

4. Drive Digital Human to Speak with Text

5. Digital Human Q&A

6. Interrupt Digital Human Speaking

​1. Initialization

​2. Connect to Digital Human

​onShow Digital human is ready to display

​onError Exception occurred when connecting to digital human

​onVideoTrack RTC media channel created successfully

​onAudioSamples Local audio sampling data callback

​onTtsSpeakStart Digital human starts playing TTS synthesized audio

​onTtsSpeakText Text content of TTS played by digital human

​onTtsSpeakStop TTS text playback completed

​onSpeakStart Digital human starts speaking

​onSpeakText Text content spoken by digital human

​onSpeakStop Digital human finished speaking

​onAsrResult ASR recognition content callback

​3. Drive Digital Human to Speak with Audio URL

​4. Drive Digital Human to Speak with Text

​5. Digital Human Q&A

​6. Interrupt Digital Human Speaking

1. Initialization

2. Connect to Digital Human

`onShow` Digital human is ready to display

`onError` Exception occurred when connecting to digital human

`onVideoTrack` RTC media channel created successfully

`onAudioSamples` Local audio sampling data callback

`onTtsSpeakStart` Digital human starts playing TTS synthesized audio

`onTtsSpeakText` Text content of TTS played by digital human

`onTtsSpeakStop` TTS text playback completed

`onSpeakStart` Digital human starts speaking

`onSpeakText` Text content spoken by digital human

`onSpeakStop` Digital human finished speaking

`onAsrResult` ASR recognition content callback

3. Drive Digital Human to Speak with Audio URL

4. Drive Digital Human to Speak with Text

5. Digital Human Q&A

6. Interrupt Digital Human Speaking