How to Ask a Multiple-Choice Question

You are studying for the CA Real Estate Exam.  You have purchased a study guide to help you through this process.  We are going to use an example from the "Unlocking the DRE Salesperson and Broker Exam, Sixth Edition", which can be purchased here.  Newer editions can be purchased on Amazon.com here.

Simulated Exam #2, Page 207, Question #93:

• A seller of property tells their broker the house is connected to the sewer. the broker relays this information to the buyer, who later finds that the house has a septic system in need of repair. the buyer would sue:
  • a. the broker and the seller.
  • b. no one, as they should have checked.
  • c. the broker only.
  • d. the owner only.

This is a multiple-choice question.  If you enter this question into the default Query Screen, you will receive a response like this:

This is a surprisingly good answer, but does not actually answer the question as posed.  Why?  Because, by default, all questions are converted into "standalone" questions before being submitted to the AI.  As the AI does not have the capacity to remember beyond the current prompt, it has to be reminded of the context of the conversation.  This is done using the "standalone" question, which is composed by combining the current question plus the conversation history. The "standalone" question is discussed in  more detail here.

As you can see in the above example, the original question has been re-worded as a standalone question which now asks "Who is responsible for the buyer's potential lawsuit regarding the false information about the house being connected to the sewer system?"  This is the essence of the original question, but certainly not a multiple choice question.

So, how do we ask a multiple-choice question to have the AI select the amongst the choices given?  

• First, you have to bring up the Advanced Query Screen as described here.  You essentially tick the "Show Advanced" checkbox and submit "New".
  
• You will now see the "Standalone Question On" checkbox on the Query Screen.  Leave it un-ticked.
  
• Now when you ask your question, it will be submitted verbatim to the AI.
  • However, note that when you turn off the "standalone" question option, the AI will no longer be able to automatically keep track of the conversation.  You only really want to turn it off in cases like this when you really need the AI to process your exact query, word for word.

Is this the correct answer?  You will have to purchase the book and look up the answer key for this question.  We do not want any copyright complaints. We used the question from that document  purely for educational purposes.

The purpose of this article is to show you how to ask a multiple-choice question successfully, if you need to.

AI is not Human

One final point.  Look at this question and response:

The AI describes the correct answer, but selects the wrong letter.  It was necessary to specifically instruct the AI to select the correct letter of the correct answer.

Now, as humans, this is intuitive. Select the letter corresponding to the correct answer.  But, we must always remember the AI is not human.  Do not leave anything to inference because the AI does not intuitively know anything.

One thing for sure you want to avoid is the "complete this sentence" sort of multiple choice question.  Why?  Because AI may have the answer to the question, but not be able to complete the sentence.

Pretend you are talking to a child in kindergarten – a genius child prodigy, for sure -- but, with dyslexia.

This is an amazing research tool, but is is important to always keep in mind the most efficient and effective techniques for using this tool.

 

Real Estate Exam Multiple Choice Question Rephrasing

The biggest problem you will find with Real Estate Study Exam questions is their reliance on "complete the sentence" type of questions.  The problem with AI is pretty obvious here:  "Complete the sentence" relies on a certain innate human ability to complete a chain of thought which Large Language Models do not possess.

Therefore a question like:

The prospective purchaser may withdraw the offer at any time before the seller's acceptance of an offer: a. unless the offer states that it is irrevocable. b. for any reason. c. provided the offeree has breached the offer. d. provided the offer is not a supported deposit.

While totally understandable to a human, the above can be downright confusing to an LLM.

Here is one way to rephrase the question in a more understandable way for a large language model:

"An offer to purchase real estate can be withdrawn by the prospective buyer at any time before it is accepted by the seller, unless which of the following conditions applies:

A) The offer states it is irrevocable

B) The buyer can withdraw for any reason

C) The seller has breached the offer

D) The offer does not include an earnest money deposit

Please select the answer choice that correctly completes the statement: An offer can be withdrawn at any time before acceptance, unless the offer states it is _______________."

The key changes are:

• Stating the overall concept first in a full sentence rather than just a sentence fragment.
• Making it clear we are talking about an offer to purchase real estate.
• Using more natural phrasing like "prospective buyer" rather than "prospective purchaser".
• Stating the answer choices as full conditions rather than sentence fragments.
• Ending with a clear question restating the main idea and asking for the selection of the right answer choice.

This frames the question in a more natural language manner that should be easier for a large language model to comprehend and answer correctly.

And, whereas the AI will fail to answer the first question correctly, 

The following, however, succeeds because it is far more specific and clear:

Now, according to the 25 Most Common Questions on the California Real Estate Exam (2023) https://www.youtube.com/watch?v=_m4lHuimlyk&t=49s&ab_channel=PrepAgent, this answer is incorrect.  The correct answer is B) The buyer can withdraw for any reason.  However, based upon the same context documents, every Large Language Model asked this question responded A), and justified it's decision based upon the context documents chosen.

You be the judge.

Here is another example:

It is difficult for the model to answer this question correctly because of the fact that not all the options are mentioned in it's context documents, and it is trained to only provide answers that it can document.  Note the change in response by adding just a couple more words:

It is still conflicted in it's answer, even thought it is the correct one, and it states why.