Dziri Voicebot: An End-to-End Low-Resource Speech-to-Speech Conversational System for Algerian Dialect

Recent breakthroughs in deep learning and large-scale pretraining, exemplified by models like DeepSpeech, Wav2Vec2, and Whisper, have significantly advanced speech recognition and NLP capabilities. These models excel in high-resource languages due to abundant annotated data, enabling applications such as virtual assistants and customer service bots. However, for low-resource languages and dialects like Algerian Darija, data scarcity and linguistic complexity pose substantial barriers. Darija's phonetic variability, frequent French code-switching, and lack of standard orthography hinder the direct application of existing models. While multilingual models like XLS-R and MARBERT have shown promise, their performance remains limited in dialectal contexts, especially under noisy conditions and with spontaneous speech. Prior efforts mainly focused on isolated tasks—either acoustic modeling or text-based NLP—without integrating these into a unified, end-to-end conversational framework. Consequently, deploying practical voice assistants in Algerian dialect remains an open challenge, necessitating innovative architectures that can handle low-resource, code-switching, and dialectal variability simultaneously.

The core challenge addressed in this work is developing a robust, end-to-end speech-to-speech system for Algerian Dialect, which is characterized by limited annotated data, high phonetic and lexical variability, and frequent code-switching with French. Existing systems are either limited to isolated modules—such as speech recognition or text understanding—or rely on high-resource language data, making them unsuitable for real-world deployment in low-resource dialectal environments. The absence of standardized orthography and the presence of multiple writing conventions further complicate the development of accurate models. Additionally, the need for natural, fluent speech synthesis in a dialect lacking dedicated TTS resources adds another layer of difficulty. Addressing these issues requires a comprehensive architecture capable of jointly handling noisy speech input, semantic interpretation, knowledge retrieval, and natural speech output, all within a resource-constrained setting.

This research introduces several key innovations:

• �� A multi-stage fine-tuning pipeline for Whisper, enabling high-accuracy ASR tailored to Algerian dialectal speech with 13.74% WER.
• �� Integration of DziriBERT embeddings into a hybrid NLU framework, achieving 98.4% intent accuracy and 93.9% entity F1-score, effectively capturing dialectal nuances.
• �� Deployment of a retrieval-augmented generation (RAG) system combining FAISS-based semantic search with Llama 3.2, supporting factual, open-domain responses.
• �� Use of LoRA techniques to adapt VITS and XTTS-v2 models for dialect-specific speech synthesis, producing natural and expressive speech.
• �� Modular system design ensuring component independence, scalability, and ease of deployment in low-resource environments.
• �� Construction of dedicated dialectal corpora for ASR and TTS, addressing the scarcity of annotated data and enabling domain-specific training.

These innovations collectively enable a comprehensive, practical speech-to-speech dialogue system tailored for Algerian dialect, overcoming traditional limitations of low-resource language processing.

• �� Data Collection: Developed a web-based platform to record speech from 14 speakers, covering 70 telecom-related intents, with annotations including transcriptions, speaker IDs, and intent labels.
• �� ASR Model Training: Fine-tuned Whisper-medium and Wav2Vec2-XLS-R models using the collected corpus, employing multi-stage training with decreasing learning rates, data augmentation (Gaussian noise, volume perturbation), and hyperparameter tuning.
• �� NLU Module: Employed DziriBERT for contextual embeddings, integrated into Rasa framework for intent classification and entity extraction, trained on 15,891 examples.
• �� Knowledge Retrieval: Built a domain-specific knowledge base, embedded documents with multilingual paraphrase models, indexed via FAISS, and used for semantic retrieval during RAG inference.
• �� Response Generation: Retrieved evidence passages fed into Llama 3.2, with prompt engineering to restrict hallucinations and ensure factual consistency.
• �� TTS System: Collected dialectal speech corpora, trained LoRA-enhanced VITS and XTTS-v2 models, optimized for naturalness and dialectal pronunciation.
• �� System Integration: Assembled all modules into a unified pipeline, ensuring smooth data flow and compatibility, enabling real-time speech-to-speech interaction.

• �� Dataset Construction: Collected 4,103 utterances, totaling 2.68 hours, with diverse speakers and linguistic variations, split into training, validation, and testing sets.
• �� Baseline Comparison: Evaluated Whisper-medium and Wav2Vec2-XLS-R models, with hyperparameters tuned for low-resource adaptation.
• �� Performance Metrics: Used Word Error Rate (WER) for ASR, intent accuracy and entity F1 for NLU, retrieval precision for RAG, and subjective naturalness scores for TTS.
• �� Ablation Studies: Assessed the impact of multi-stage fine-tuning, data augmentation, and retrieval strategies on overall system performance.
• �� Cross-Scenario Testing: Evaluated robustness under noisy conditions and with unseen speakers, demonstrating system resilience.
• �� Hyperparameter Optimization: Fine-tuned learning rates, batch sizes, and model architectures to maximize accuracy and efficiency.

• �� The Whisper-based ASR achieved a WER of 13.74%, outperforming Wav2Vec2-XLS-R, which achieved 17.2%, validating the effectiveness of transfer learning and multi-stage fine-tuning.
• �� NLU modules achieved 98.4% intent classification accuracy and 93.9% entity recognition F1-score, surpassing previous benchmarks in dialectal Arabic NLP.
• �� The RAG system scored 78.5/100 in open-domain QA, demonstrating effective retrieval and factual consistency.
• �� TTS models generated highly natural speech, with subjective evaluations rating naturalness above 4.2/5.
• �� The integrated system demonstrated end-to-end speech-to-speech interaction with low latency and high robustness, suitable for real-world deployment.

• �� The system can be deployed in Algerian telecom customer service centers, providing automated voice assistants capable of understanding and responding in Darija.
• �� It supports multilingual environments, handling code-switching seamlessly, thus improving accessibility for users who mix Arabic and French.
• �� Educational platforms can leverage the system for dialect learning and preservation, promoting local language use.
• �� The architecture can be adapted for other North African dialects, fostering inclusive speech technology development.
• �� Industry-wise, it reduces operational costs and enhances user engagement through natural, dialectal speech interactions.

• �� The system’s robustness under noisy, multi-speaker, or highly accented conditions remains limited, requiring further data augmentation and model robustness enhancements.
• �� Recognition accuracy for rare or extreme dialectal pronunciations needs improvement, necessitating larger and more diverse datasets.
• �� Computational complexity and resource demands are high, especially for real-time deployment on edge devices, calling for model compression and optimization.