5  Answer Generation

(ns notebooks.generation
  (:require
   [clojure.string :as str]
   [notebooks.llm-api :as llm]
   [notebooks.preparation :refer [ds]]
   [scicloj.kindly.v4.kind :as kind]
   [notebooks.vdb-evaluation :as vdb]
   [tablecloth.api :as tc]))

To generate an answer using a RAG approach, all we have to do is add our retrieved context to a prompt.

To start with, we will re-use the same question as we used in the previous section.

(def sample-question "How much annual investment was provided under the 2019 GP agreement?")

Next, we will create some prompt functions to help build the context.

5.1 Context Prompt

(defn add-context [{:keys [question] :as rag-data} db-store-name]
  (let [ctx (mapv :text (vdb/generate-context question db-store-name))]
    (assoc rag-data :retrieved-context ctx)))

(defn add-pq-prompt [{:keys [retrieved-context] :as rag-data}]
  (assoc rag-data :system-prompt
         (str "I want you to act as a responsible and trustworthy senior government official.
Please provide an answer to a citizen's question, using only the context provided.
Answer as if you are talking directly to the citizen and be neutral and formal as possible.
If you can't find a specific detail from the question, please acknowledge this and provide any
other helpful information that may be related to the question.
If you can't find sufficient information in the context to answer the question at all,
then reply with \"I am unable to answer this question with the information I have available.\""
              "\n\n CONTEXT: " (str/join "\n\n" retrieved-context))))

(defn quoted-response
  "To help distinuish the llm text from other text in the notebook"
  [text]
  (->>
   (str/split-lines text)
   (mapv (fn [line] (str "> " line)))
   (str/join "\n")))

(kind/md
 (-> {:question sample-question}
     (add-context :db-docs)
     add-pq-prompt
     :system-prompt
     quoted-response))

I want you to act as a responsible and trustworthy senior government official. Please provide an answer to a citizen’s question, using only the context provided. Answer as if you are talking directly to the citizen and be neutral and formal as possible. If you can’t find a specific detail from the question, please acknowledge this and provide any other helpful information that may be related to the question. If you can’t find sufficient information in the context to answer the question at all, then reply with “I am unable to answer this question with the information I have available.”

CONTEXT: Following it’s completion a paper is to be presented to me outlining the findings of the review and setting out recommended actions to respond to the identified findings. It is worth noting that the Government has undertaken measures to increase GP capacity and improved access to services. Increased annual investment of over €211 million was provided for under the 2019 GP Agreement, providing for increased GP fees, supports and new services.

The 2019 GP Agreement provided for increased annual investment of over €211 million for general practice, providing for increased GMS GP fees, supports and new services. In particular the 2019 Agreement increased the enhanced supports package for rural GP practices by 10%. A further €30 million has been provided for under the GP Agreement 2023 for practice supports to expand and retain practice staff.

The 2019 GP Agreement provided for increased annual investment of over €211 million for general practice, providing for increased GMS GP fees, supports and new services. A further €30 million has been provided for under the GP Agreement 2023 for practice supports to expand and retain practice staff. The annual intake of doctors into the GP training programme has been increased by 80% from 2015 to 2023; there were 286 new entrants to GP training in 2023 and it is planned to increase the number of places for new entrants to 350 for this year.

The Government is aware of the workforce issues currently facing general practice, including the limited access to GP services in certain areas, and is working to ensure that general practice is sustainable in all areas into the future. Under the 2019 GP Agreement additional annual expenditure provided for general practice has been increased now by €211.6m. This provides for significant increases in capitation fees for participating GMS GPs, and new fees and subsidies for additional services.

Under the 2019 GP Agreement additional annual expenditure provided for general practice was increased by €211.6m. This provided for significant increases in capitation fees for participating GMS GPs, and new fees for additional services and increased practice supports. The GP Agreement 2023 further increased GP capitation fees, increased the existing subsidy rates for practice staff, and introduced a grant support for additional staff capacity as well a practice staff maternity leave support.

5.2 LLM Answer Generation

Now that we have a way to generate a prompt, let’s pass it to an LLM.

(defn add-llm-response [{:keys [model-ref question system-prompt] :as rag-data}]
  (let [answer (llm/ask-llm
                {:model-ref model-ref
                 :question question
                 :system-prompt system-prompt})]
    (assoc rag-data :answer answer)))

(comment
  (spit "data/generation_examples/example_1.txt"
        (-> {:question sample-question
             :model-ref "gpt-3.5-turbo"}
            (add-context :db-docs)
            add-pq-prompt
            add-llm-response
            :answer)))

(comment
  (-> {:question sample-question
       :model-ref "gpt-3.5-turbo"}
      (add-context :db-docs)
      add-pq-prompt
      add-llm-response
      :answer
      (kind/md)))

(-> "data/generation_examples/example_1.txt"
    slurp
    quoted-response
    kind/md)

Under the 2019 GP Agreement, an increased annual investment of over €211 million was provided for general practice. This funding was allocated to increase GMS GP fees, supports, and new services in order to enhance general practice services across the country. Additionally, the Government has allocated a further €30 million under the GP Agreement 2023 to support practice staff expansion and retention efforts.

Let’s try a few additional tests:

(def sample-question-broad "What is the government doing to support GPs?")

(def sample-question-detail "What is the government doing to support GPs in Limerick city?")

(comment
  (spit "data/generation_examples/example_2.txt"
        (-> {:question sample-question-broad
             :model-ref "gpt-3.5-turbo"}
            (add-context :db-docs)
            add-pq-prompt
            add-llm-response
            :answer))

  (spit "data/generation_examples/example_3.txt"
        (-> {:question sample-question-detail
             :model-ref "gpt-3.5-turbo"}
            (add-context :db-docs)
            add-pq-prompt
            add-llm-response
            :answer)))

(-> "data/generation_examples/example_2.txt"
    slurp
    quoted-response
    kind/md)

I appreciate your inquiry. The Government is actively addressing the challenges facing general practice to ensure sustainable healthcare services for all citizens. Steps have been taken to enhance GP training opportunities and increase the number of practicing GPs in the country to improve access to care. Additionally, a comprehensive Strategic Review of General Practice is underway, involving input from various stakeholders to address issues related to GP capacity and develop measures for a more sustainable future for general practice. This review will also focus on attracting GPs to underserved rural and urban areas to improve access to healthcare services in those regions.

(-> "data/generation_examples/example_3.txt"
    slurp
    quoted-response
    kind/md)

I appreciate your inquiry regarding the support for GPs in Limerick city. The government recognizes the vital role of GPs in primary care and has been actively working to address workforce challenges and improve access to GP services across various areas, including Limerick.

Efforts have been made to enhance GP training opportunities, and the sustainability of general practice is a key focus for the government. While GPs are independent contractors and can establish practices where they choose, the government is committed to ensuring that general practice remains viable in all areas, including those in Limerick city.

Moreover, a review involving key stakeholders is underway to address issues affecting general practice, such as GP capacity and access to services in underserved areas. This review aims to identify necessary measures to promote a sustainable future for general practice, which will benefit practitioners and patients alike.

If you have any specific concerns or suggestions regarding GP services in Limerick city, please feel free to share them, as your feedback is valuable in shaping ongoing efforts to enhance healthcare provision in the region. Thank you for your query.

Putting these together into single function:

(defn get-rag-answer [rag-data db-store-name]
  (-> rag-data
      (add-context db-store-name)
      add-pq-prompt
      add-llm-response))

5.3 Answer with References

As an added feature, we could use the original dataset to try provide a reference for the question.

(defn get-reference-link-for-doc [doc]
  (-> ds
      (tc/drop-missing :answer)
      (tc/select-rows #(re-find (re-pattern doc) (:answer %)))
      :url
      first))

(defn format-links-md [links]
  (str "*References* \n\n- "
       (str/join "\n\n- "
                 links)))

(defn generate-answer-with-references [rag-data db-store-name]
  (let [rag-data (get-rag-answer rag-data db-store-name)
        ctx-docs (:retrieved-context rag-data)
        docs-ref-links (format-links-md (mapv get-reference-link-for-doc ctx-docs))]
    (str (:answer rag-data) "\n\n" docs-ref-links)))

(comment
  (spit "data/generation_examples/example_4.txt"
        (-> {:question sample-question
             :model-ref "gpt-3.5-turbo"}
            (generate-answer-with-references :db-docs))))

(-> "data/generation_examples/example_4.txt"
    slurp
    quoted-response
    (kind/md))

Under the 2019 GP Agreement, an increased annual investment of over €211 million was provided for general practice. This investment facilitated increased GMS GP fees, supports, and the introduction of new services. Additionally, the 2019 Agreement allocated a further €30 million under the GP Agreement 2023 for practice supports aimed at expanding and retaining practice staff.

References

• https://www.oireachtas.ie/en/debates/question/2024-01-18/60/

• https://www.oireachtas.ie/en/debates/question/2024-01-18/316/

• https://www.oireachtas.ie/en/debates/question/2024-01-18/337/

• https://www.oireachtas.ie/en/debates/question/2024-01-31/141/

• https://www.oireachtas.ie/en/debates/question/2024-02-29/59/

source: src/notebooks/generation.clj