finish voice chat
This commit is contained in:
zachary62
@@ -53,28 +53,31 @@ flowchart TD
|
||||
> 2. Include only the necessary utility functions, based on nodes in the flow.
|
||||
|
||||
1. **`record_audio()`** (`utils/audio_utils.py`)
|
||||
- *Input*: (Optional) `silence_threshold` (float, e.g., RMS energy), `min_silence_duration_ms` (int), `chunk_size_ms` (int), `sample_rate` (int, Hz), `channels` (int).
|
||||
- *Output*: A tuple `(audio_data, sample_rate)` where `audio_data` is in-memory audio (e.g., bytes or NumPy array) and `sample_rate` is the recording sample rate (int).
|
||||
- *Description*: Records audio from the microphone. Starts recording when sound is detected above `silence_threshold` (optional, or starts immediately) and stops after `min_silence_duration_ms` of sound below the threshold.
|
||||
- *Input*: (Optional) `sample_rate` (int, Hz, e.g., `DEFAULT_SAMPLE_RATE`), `channels` (int, e.g., `DEFAULT_CHANNELS`), `chunk_size_ms` (int, e.g., `DEFAULT_CHUNK_SIZE_MS`), `silence_threshold_rms` (float, e.g., `DEFAULT_SILENCE_THRESHOLD_RMS`), `min_silence_duration_ms` (int, e.g., `DEFAULT_MIN_SILENCE_DURATION_MS`), `max_recording_duration_s` (int, e.g., `DEFAULT_MAX_RECORDING_DURATION_S`), `pre_roll_chunks_count` (int, e.g., `DEFAULT_PRE_ROLL_CHUNKS`).
|
||||
- *Output*: A tuple `(audio_data, sample_rate)` where `audio_data` is a NumPy array of float32 audio samples, and `sample_rate` is the recording sample rate (int). Returns `(None, sample_rate)` if no speech is detected or recording fails.
|
||||
- *Description*: Records audio from the microphone using silence-based Voice Activity Detection (VAD). Buffers `pre_roll_chunks_count` of audio and starts full recording when sound is detected above `silence_threshold_rms`. Stops after `min_silence_duration_ms` of sound below the threshold or if `max_recording_duration_s` is reached.
|
||||
- *Necessity*: Used by `CaptureAudioNode` to get user\'s voice input.
|
||||
|
||||
2. **`speech_to_text_api(audio_data, sample_rate)`** (`utils/speech_to_text.py`)
|
||||
- *Input*: `audio_data` (bytes or NumPy array), `sample_rate` (int).
|
||||
- *Input*: `audio_data` (bytes), `sample_rate` (int, though the API might infer this from the audio format).
|
||||
- *Output*: `transcribed_text` (str).
|
||||
- *Necessity*: Used by `SpeechToTextNode` to convert in-memory audio data to text.
|
||||
- *Example Model*: OpenAI `gpt-4o-transcribe`.
|
||||
|
||||
3. **`call_llm(prompt, history)`** (`utils/llm_service.py`)
|
||||
- *Input*: `prompt` (str), `history` (list of dicts, e.g., `[{"role": "user", "content": "..."}]`)
|
||||
3. **`call_llm(messages)`** (`utils/call_llm.py`)
|
||||
- *Input*: `messages` (list of dicts, e.g., `[{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]`). This should be the complete conversation history including the latest user query.
|
||||
- *Output*: `llm_response_text` (str)
|
||||
- *Necessity*: Used by `QueryLLMNode` to get an intelligent response.
|
||||
- *Example Model*: OpenAI `gpt-4o`.
|
||||
|
||||
4. **`text_to_speech_api(text_to_synthesize)`** (`utils/text_to_speech.py`)
|
||||
- *Input*: `text_to_synthesize` (str).
|
||||
- *Output*: A tuple `(audio_data, sample_rate)` where `audio_data` is in-memory audio (e.g., NumPy array) and `sample_rate` is the audio sample rate (int).
|
||||
- *Output*: A tuple `(audio_data, sample_rate)` where `audio_data` is in-memory audio as bytes (e.g., MP3 format from OpenAI) and `sample_rate` is the audio sample rate (int, e.g., 24000 Hz for OpenAI `gpt-4o-mini-tts`).
|
||||
- *Necessity*: Used by `TextToSpeechNode` to convert LLM text to speakable in-memory audio data.
|
||||
- *Example Model*: OpenAI `gpt-4o-mini-tts`.
|
||||
|
||||
5. **`play_audio_data(audio_data, sample_rate)`** (`utils/audio_utils.py`)
|
||||
- *Input*: `audio_data` (NumPy array), `sample_rate` (int).
|
||||
- *Input*: `audio_data` (NumPy array of float32 audio samples), `sample_rate` (int).
|
||||
- *Output*: None
|
||||
- *Necessity*: Used by `TextToSpeechNode` (in its `post` method) to play the in-memory synthesized speech.
|
||||
|
||||
@@ -88,11 +91,8 @@ The shared memory structure is organized as follows:
|
||||
|
||||
```python
|
||||
shared = {
|
||||
"user_audio_data": None, # In-memory audio data (bytes or NumPy array) from user
|
||||
"user_audio_data": None, # In-memory audio data (NumPy array) from user
|
||||
"user_audio_sample_rate": None, # int: Sample rate of the user audio
|
||||
"user_text_query": None, # str: Transcribed user text
|
||||
"llm_text_response": None, # str: Text response from LLM
|
||||
# "llm_audio_data" and "llm_audio_sample_rate" are handled as exec_res within TextToSpeechNode's post method
|
||||
"chat_history": [], # list: Conversation history [{"role": "user/assistant", "content": "..."}]
|
||||
"continue_conversation": True # boolean: Flag to control the main conversation loop
|
||||
}
|
||||
@@ -107,40 +107,41 @@ shared = {
|
||||
- *Type*: Regular
|
||||
- *Steps*:
|
||||
- *prep*: Check `shared["continue_conversation"]`. (Potentially load VAD parameters from `shared["config"]` if dynamic).
|
||||
- *exec*: Call `utils.audio_utils.record_audio()` (passing VAD parameters if configured).
|
||||
- *post*: `audio_data, sample_rate = exec_res`. Write `audio_data` to `shared["user_audio_data"]` and `sample_rate` to `shared["user_audio_sample_rate"]`. Returns `"default"`.
|
||||
- *exec*: Call `utils.audio_utils.record_audio()` (passing VAD parameters if configured). This returns a NumPy array and sample rate.
|
||||
- *post*: `audio_numpy_array, sample_rate = exec_res`. Write `audio_numpy_array` to `shared["user_audio_data"]` and `sample_rate` to `shared["user_audio_sample_rate"]`. Returns `"default"`.
|
||||
|
||||
2. **`SpeechToTextNode`**
|
||||
- *Purpose*: Convert the recorded in-memory audio to text.
|
||||
- *Type*: Regular
|
||||
- *Steps*:
|
||||
- *prep*: Read `shared["user_audio_data"]` and `shared["user_audio_sample_rate"]`. Return `(user_audio_data, user_audio_sample_rate)`.
|
||||
- *exec*: `audio_data, sample_rate = prep_res`. Call `utils.speech_to_text.speech_to_text_api(audio_data, sample_rate)`.
|
||||
- *prep*: Read `shared["user_audio_data"]` (NumPy array) and `shared["user_audio_sample_rate"]`. Return `(user_audio_data_numpy, user_audio_sample_rate)`.
|
||||
- *exec*: `audio_numpy_array, sample_rate = prep_res`. **Convert `audio_numpy_array` to audio `bytes` (e.g., in WAV format using `scipy.io.wavfile.write` to an `io.BytesIO` object).** Call `utils.speech_to_text.speech_to_text_api(audio_bytes, sample_rate)`.
|
||||
- *post*:
|
||||
- Write `exec_res` (transcribed text) to `shared["user_text_query"]`.
|
||||
- Append `{"role": "user", "content": exec_res}` to `shared["chat_history"]`.
|
||||
- Let `transcribed_text = exec_res`.
|
||||
- Append `{"role": "user", "content": transcribed_text}` to `shared["chat_history"]`.
|
||||
- Clear `shared["user_audio_data"]` and `shared["user_audio_sample_rate"]` as they are no longer needed.
|
||||
- Returns `"default"`.
|
||||
- Returns `"default"` (assuming STT is successful as per simplification).
|
||||
|
||||
3. **`QueryLLMNode`**
|
||||
- *Purpose*: Get a response from the LLM based on the user\'s query and conversation history.
|
||||
- *Purpose*: Get a response from the LLM based on the user's query and conversation history.
|
||||
- *Type*: Regular
|
||||
- *Steps*:
|
||||
- *prep*: Read `shared["user_text_query"]` and `shared["chat_history"]`. Return `(user_text_query, chat_history)`.
|
||||
- *exec*: Call `utils.llm_service.call_llm(prompt=prep_res[0], history=prep_res[1])`.
|
||||
- *prep*: Read `shared["chat_history"]`. Return `chat_history`.
|
||||
- *exec*: `history = prep_res`. Call `utils.call_llm.call_llm(messages=history)`.
|
||||
- *post*:
|
||||
- Write `exec_res` (LLM text response) to `shared["llm_text_response"]`.
|
||||
- Append `{"role": "assistant", "content": exec_res}` to `shared["chat_history"]`.
|
||||
- Returns `"default"`.
|
||||
- Let `llm_response = exec_res`.
|
||||
- Append `{"role": "assistant", "content": llm_response}` to `shared["chat_history"]`.
|
||||
- Returns `"default"` (assuming LLM call is successful).
|
||||
|
||||
4. **`TextToSpeechNode`**
|
||||
- *Purpose*: Convert the LLM\'s text response into speech and play it.
|
||||
- *Purpose*: Convert the LLM's text response into speech and play it.
|
||||
- *Type*: Regular
|
||||
- *Steps*:
|
||||
- *prep*: Read `shared["llm_text_response"]`.
|
||||
- *exec*: Call `utils.text_to_speech.text_to_speech_api(prep_res)`. This returns `(llm_audio_data, llm_sample_rate)`.
|
||||
- *post*: `llm_audio_data, llm_sample_rate = exec_res`.
|
||||
- Call `utils.audio_utils.play_audio_data(llm_audio_data, llm_sample_rate)`.
|
||||
- *prep*: Read `shared["chat_history"]`. Identify the last message, which should be the LLM's response. Return its content.
|
||||
- *exec*: `text_to_synthesize = prep_res`. Call `utils.text_to_speech.text_to_speech_api(text_to_synthesize)`. This returns `(llm_audio_bytes, llm_sample_rate)`.
|
||||
- *post*: `llm_audio_bytes, llm_sample_rate = exec_res`.
|
||||
- **Convert `llm_audio_bytes` (e.g., MP3 bytes from TTS API) to a NumPy array of audio samples (e.g., using a library like `pydub` or `soundfile` to decode).**
|
||||
- Call `utils.audio_utils.play_audio_data(llm_audio_numpy_array, llm_sample_rate)`.
|
||||
- (Optional) Log completion.
|
||||
- If `shared["continue_conversation"]` is `True`, return `"next_turn"` to loop back.
|
||||
- Otherwise, return `"end_conversation"`.
|
||||
|
||||
Reference in New Issue
Block a user