The Real Estate Books AI Library system utilizes Artificial Intelligence Large Language Models (LLMs) to answer queries. Currently, we utilize OpenAI models GPT-4 (Gold) and GPT-3.5-Turbo-16K (Bronze). We are currently testing a 3rd model, our Silver model, but at the time of this writing, we are under an NDA and cannot reveal the provider.
Attached to this post are the results of several actual California Real Estate Practice Exams taken by our AI systems utilizing only the documents posted in this library. These are the full exam questions entered and actual responses received.
So far, with all CA Real Estate Exam questions posed to our AI, it has a 90% - 95% success rate. Please note that this rate is affected by these elements:
- Available context documents. All questions are answered using only the context documents available in this library. The models are trained to ONLY use these documents. This eliminates the possibility of "hallucination". But it also means that the system cannot respond to questions that refer to content (names, places, things) that are NOT in the available documentation.
- Question Phraseology. We have chosen primarily practice exam questions because they are a reliable source for reliable answers against which we can test the efficacy of our system. These exam questions appear to always be multiple choice, which is the worst format for an AI system because it depends on inference of the the meaning of the question by "completing the sentence". Current AI technology does not possess the innate ability to "infer" like humans or critically analyze like humans. So, sometimes the AI will not be able to answer the question, not because the answer is not in it's context library, but because it cannot clearly understand the question.
We hope in the future to mitigate these issues by using the technique of "fine-tuning", that is, training the AI on the log history of actual questions and responses processed. That way, over time, system will become smarter and smarter, and will learn to understand all the various ways the same question can be asked.
That said, please review the attached exam questions and responses. Note how the AI responds to each question, listing it's source citations and basing it's conclusions on the available context documents.
