We ensure that Nova is reliable and effective through a comprehensive validation process that includes both past and ongoing processes.

Past Validation

  • Quality Control and Improvement:
    • Our team, including Data Scientists, Clinical Psychologists, and Product Developers, specially designed guiding instructions (known as a system prompt) for Nova to ensure the model navigates its knowledge base efficiently, delivering accurate, safe, and relevant responses.
    • Before release, the system prompt was rigorously tested by the team and compared against our established safety and performance test cases to ensure Nova behaved in the intended manner.
  • Prompt Engineering and Testing:
    • We established a structured process for making changes to Nova’s system prompt. Before deployment, these instructions underwent rigorous testing by the team and stakeholders using a combination of predefined scenarios, unscripted exploratory tests, and evaluation criteria to ensure Nova’s system prompt was safe and effective.
  • Foundation Model Validation:
    • Nova is built on OpenAI foundation models, which have undergone extensive validation. Currently, Nova works on GPT-5 which has extended their track record of safe and validated model development, including incorporating feedback from over 50 experts in various domains and using reinforcement learning with human feedback (RLHF) to fine-tune its behaviour.

Ongoing Validation

  • Quality Control and Improvement:
    • Our team continues to review pseudonymized conversation transcripts regularly to identify new issues and make necessary adjustments. Any behavioural issues or bugs are addressed promptly through system prompt changes.
    • Improvements and fixes are also informed by user feedback to ensure commitment to continuous improvement keeps the chatbot aligned with user needs and technological advancements.
  • Prompt Engineering and Testing:
    • We maintain our structured process for modifying Nova’s guiding instructions. Before any updates are deployed, they undergo rigorous testing to ensure continued safety and effectiveness.
  • Content Moderation and Guided Responses:
    • We enforce strict content moderation to prevent the chatbot from engaging in inappropriate topics. In these instances, we still provide users with helpful information through hard-coded messages directing them to appropriate resources (compared with other models that end the conversation without providing resources).