Generate Follow-Up Questions
Models can sometimes correctly answer sub-problems but incorrectly answer the overall query. This is known as the compositionality gap1.
How can we encourage a model to use the answers to sub-problems to correctly generate the overall solution?
Self-Ask is a technique which use a single prompt to:
- decide if follow-up questions are required
- generate the follow-up questions
- answer the follow-up questions
- answer the main query
Implementation¶
import instructor
from openai import OpenAI
from pydantic import BaseModel, Field
client = instructor.from_openai(OpenAI())
class FollowUp(BaseModel):
question: str = Field(description="The follow-up question")
answer: str = Field(description="The answer to the follow-up question")
class Response(BaseModel):
follow_ups_required: bool
follow_ups: list[FollowUp]
final_answer: str
def self_ask(query):
return client.chat.completions.create(
model="gpt-4o",
response_model=Response,
messages=[
{
"role": "system",
"content": f"""Query: {query}
Are follow-up questions needed?
If so, generate follow-up questions, their answers, and then the final answer to the query.
""", # !(1)
},
],
)
if __name__ == "__main__":
query = "Who was president of the U.S. when superconductivity was discovered?"
response = self_ask(query)
print(response.follow_ups_required)
#> True
for follow_up in response.follow_ups:
print(follow_up)
"""
question='When was superconductivity discovered?' answer='Superconductivity was discovered in April 1911.'
"""
"""
question='Who was president of the U.S. in April 1911?' answer='William Howard Taft was the President of the United States in April 1911.'
"""
print(response.final_answer)
"""
William Howard Taft was president of the U.S. when superconductivity was discovered.
"""
- Without
instructor
, this prompt would generally be implemented as a one-shot or few-shot prompt1 to encourage thinking through follow-up questions. Withinstructor
, we use a zero-shot prompt!
References¶
1: Measuring and Narrowing the Compositionality Gap in Language Models