As artificial intelligence becomes more integrated into business operations and everyday life, it’s important that companies are aware of potential risks and compliant with local laws in the markets where the technology is used. Failure to do so can result in severe consequences, including legal actions and hefty fines.

Llama first drew the SAIF CHECK team’s attention when they read the 2023 paper published by the Llama 2 team. Hussain says they were especially intrigued by the Llama team’s approach to solving a common problem with conversational AI systems—they tend to lose track of context over the course of a conversation. For example, if you tell an AI model to respond only in haiku, it may forget that initial instruction after a couple of conversational turns unless you repeat it with every new request. Having to repeat the instruction takes up valuable tokens and limits the overall length of the conversation.

To address this issue, the Llama team developed a training technique called Ghost Attention (GAtt), which uses reinforcement learning with human feedback to fine-tune the model’s responses, keeping the initial instruction in mind. This results in an AI model that is much better at retaining initial instructions over the course of a multi-turn conversation.

“Because our AI Model evaluation surveys are processed over multiple runs, we take advantage of Llama’s GAtt mechanism, which helps control dialogue flow over multiple turns,” says Hussain. “By doing so, our platform can offer users more precise and informative responses to raise the quality of the output from our services.”

To customize Llama for their use case, SAIF CHECK has configured multiple layers through an additive fine-tuning process. Using Llama 3 Instruct, the generation layer receives a person’s prompt and context. Its outputs are fed into a regulatory classifier trained on various regulatory bodies and country-specific regulatory documents from SAIF CHECK’s comprehensive knowledge base. This enables the model to categorize the prompt and context within a distinct country and regulatory body.

After learning more about the responsible AI principles used to train Llama, the team decided to switch to using Llama models for text generation. Meta has made significant efforts to both blue-team and red-team its Llama models, which gives the SAIF CHECK team confidence that the Llama models align with their own priorities.

“By using Llama, we’re using an ethically trained and sourced model in the core of our process, so our process aligns with our values,” says Hussain.

She acknowledges that challenges remain in figuring out how to correctly target documents, query them, and generate responses that are appropriate to each person’s context and specific requirements.

“Every machine learning model is different, and each company deploys their model with a unique process,” she adds.

Hussain is confident that the team’s approach, which involves “chunking” documents into smaller pieces of content, will be successful.

“We believe Llama is an excellent model to use to validate our hypothesis around chunking strategies and monitor the effectiveness of our services’ responses,” she says.

The responsible grounding of Llama and its transparency are crucial to the team’s values and their view of how AI can make a global impact. SAIF CHECK believes that the true role of AI will be in complementing and enhancing humans’ use of computers.

For that to work, humans need to trust the AI models they’re using. That trust is an essential building block for SAIF CHECK—both for its own AI models and the models it validates for customers.

“Since Llama is open source, we can literally see its development, trust its documentation, and have confidence that we’re not alone in understanding and implementing this model into real-world services,” says Hussain.