4 Retrieval Evaluation
(ns notebooks.vdb-evaluation
(:require [clojure.edn :as edn]
[notebooks.preparation :refer [ds]]
[notebooks.vector-database :refer [add-doc-to-store db-store-questions query-db-store]]
[scicloj.kindly.v4.kind :as kind]
[notebooks.tokenizer :as tokenizer]
[scicloj.tableplot.v1.plotly :as plotly]
[clojure.string :as str]
[tablecloth.api :as tc]
[notebooks.vdb-evaluation :as vdb]
[jsonista.core :as json])
(:import
(dev.langchain4j.data.segment TextSegment)
(dev.langchain4j.store.embedding.inmemory InMemoryEmbeddingStore)
(dev.langchain4j.model.embedding.onnx.allminilml6v2 AllMiniLmL6V2EmbeddingModel)))In this section we’ll use some standard approaches to try to evaluate the performance of the vector database.
The material fetched from the database makes up the context that is provided in a RAG application, so it is important to try get information that is both relevant to the question, but also doesn’t contain too much superfluous information that might confuse the LLM or make the context information ambiguous.
4.1 Sample Answers and Questions
For this exercise, we’ll first create a small sample of answers (‘highlights’) and questions that we will use to test the performance of the retrieval method.
(def highlights-answers
["Under the 2019 GP Agreement additional annual expenditure provided for general practice has been increased now by €211.6m."
"As per HSE eligibility criteria, the educational requirement for a Health Care Assistant is the relevant Health Skills Level 5 (QQI) qualification."
"The national drug strategy, Reducing Harm, Supporting Recovery, sets out government policy on drug and alcohol use for the period 2017 to 2025."
"However, local authorities were invited to submit up to 5 applications to the value of €1.5 million per local authority."
"The salary scale for an archaeologist in the local government sector ranges from €55,519 to €77,176. "])(def highlights-questions
["What is the government doing to help improve GP services?"
"Will the government put in place Level 6 (QQI) courses for healthcare assistants?"
"What is the government doing with regard to the National Drugs Strategy?"
"How is the government encouraging local authorities to apply for the town and village renewal scheme?"
"What is the salary scale for an archaeologist in the local government sector?"])4.2 Evaluating Retrieval
At the moment, our approach is based on searching for similar questions, and then returning their answers. However, this is a very naive approach. It is based on the concrete steps that are usually taken when trying to answer a new question:
Search for previous similar questions
Scan these answers for relevant info
Why not just skip this step of searching through questions all together? The approach chosen here might depend on the actual application of a system like this. For example, would the intended use be to help administers prepare answers to new questions? In this case searching through previous questions might be more useful. If the intended use is something closer to ‘general’ retrieval of information, then the best approach might be to simply search through all previous answers for the info.
In order to determine the most optimal approach, there are some metrics we could introduce to test the performance of different methods.
As a starting point, let’s see what kind of information the system we’ve built so far returns. We’ll compare these responses to a more optimized approach at the end.
(->
(query-db-store "What is the government doing to help improve GP services?" 5)
(tc/dataset)
(tc/map-columns :answer [:text] (fn [t] (-> ds
(tc/select-rows #(= t (:question %)))
:answer
first)))
(kind/table))| text | score | answer |
|---|---|---|
| Deputy Michael Ring asked the Minister for Public Expenditure, National Development Plan Delivery and Reform if his Department will sanction an increase for GP's who are in receipt of district medical officer salaries in view of the immense service that they provide in rural areas; and if he will make a statement on the matter. | 0.7787127204486028 | This is a matter for the Department of Health. |
| Deputy Aindrias Moynihan asked the Minister for Health what measures are being taken to ensure adequate GP services available in County Cork; and if he will make a statement on the matter. | 0.7664299740546427 | GPs are self-employed practitioners and therefore may establish practices at a place of their own choosing. There is no prescribed ratio of GPs to patients and the State does not regulate the number of GPs that can set up in a town or community. Under the GMS scheme, the HSE contracts GPs to provide medical services without charge to medical card and GP visit card holders. Currently, there are 2,518 GPs contracted to provide services under the GMS Scheme. Where a vacancy arises in a practice with a GMS contract, the HSE becomes actively involved in the recruitment process to find a replacement GP. As of the 1st December last, there are two GMS GP vacancies in County Cork, one in Cork City and the other in Youghal. For GMS patients unable to locate a GP themselves, the Health Service Executive may assign an eligible person to be included on a medical practitioner's GMS list, in accordance with the GMS contract. The Government is aware of the workforce issues currently facing general practice, including the limited access to GP services in certain areas, and is working to ensure that general practice is sustainable in all areas into the future. Under the 2019 GP Agreement additional annual expenditure provided for general practice has been increased now by €211.6m. This provides for significant increases in capitation fees for participating GMS GPs, and new fees and subsidies for additional services. Improvements to GP’s maternity and paternity leave arrangements and a support for GPs in disadvantaged urban areas, have also been provided for. In addition, the enhanced supports package for rural GP practices was increased by 10%. The GP Agreement 2023, which provided for the expansions of GP care without charges to those who earn up to the median household income and to children aged 6 & 7, includes additional capacity supports to enable the expansion and retention of staffing within general practice. It includes additional supports for GP Out of Hours services also. These measures help make general practice in Ireland a more attractive career choice for doctors and will see an increase in the number of GPs working in the State, improving access to GP services for patients throughout the country. Figures recently released by the ICGP show that 1,311 medical graduates have applied for GP training in 2024, a notable increase on last year and higher than any previous years. The number of doctors entering GP training has been increased significantly in recent years, with 286 new entrants in 2023 and 350 places planned for new entrants for this year. Annual intake to the GP training scheme has been increased by over 80% from 2015 to 2023, and the number of new entrant places to be available this year is a 22% increase on last year's intake alone. Furthermore, GP recruitment is ongoing under the joint non-EU GP Training Programme between the HSE and ICGP. It was planned to bring up to 100 non-EU GPs to Ireland in 2023, and it is now intended to bring to Ireland up to 250 more non-EU GPs by the end of 2024. The programme will help to quickly improve access to GP services, particularly in areas with limited access. Lastly, my Department and the HSE have commenced a Strategic Review of General Practice. The review, with input from key stakeholders, is examining the broad range of issues affecting general practice, including issues related to GP capacity, and when completed will set out the measures necessary to deliver a more sustainable general practice into the future. |
| Deputy David Cullinane asked the Minister for Health his plans to increase access to rural GP and local health care services; to report on his engagements with his Department and Health Service Executive officials and an organisation (details supplied) regarding the development of a directly employed GP contract; and if he will make a statement on the matter. | 0.7538880260414211 | GPs are self-employed practitioners and therefore may establish practices at a place of their own choosing. There is no prescribed ratio of GPs to patients and the State does not regulate the number of GPs that can set up in a town or community. The Government is working to increase the number of GPs practicing in the State and thereby improve access to GP services across the country. Several measures have been taken to increase GP capacity, including increasing investment in general practice to support practices and make general practice a more attractive career choice, increasing the number of doctors training to become GPs, and establishing the non-EU GP Training Programme to rapidly bring more GPs to Ireland. Under the 2019 GP Agreement additional annual expenditure provided for general practice was increased by €211.6m. This provided for significant increases in capitation fees for participating GMS GPs, and new fees for additional services and increased practice supports. The GP Agreement 2023 further increased GP capitation fees, increased the existing subsidy rates for practice staff, and introduced a grant support for additional staff capacity as well a practice staff maternity leave support. An enhanced supports package for rural GP practices was introduced previously to support rural GPs, these supports were increased by 10% under the 2019 GP Agreement. Practices in receipt of rural practice supports attract the maximum allowable rates for practice staff support subsidies and locum contributions for leave taking. The number of doctors entering GP training has been increased significantly in recent years, with 286 new entrants in 2023 and 350 places planned for new entrants for this year. Annual intake to the GP training scheme has been increased by over 80% from 2015 to 2023, and the number of new entrant places to be available this year is a 22% increase on last year's intake. As a result of these increases, it is estimated over the next few years that 3 to 6 GP graduates will on-board for every 2 GMS retirements. GP recruitment is ongoing under the joint non-EU GP Training Programme between the HSE and ICGP. 112 non-EU GPs were recruited last year under the training programme, of which 84 were in practice by the end of January. I have secured funding to recruit up to 250 more non-EU GPs to Ireland this year. The placement of GPs under the programme is targeted to rural and underserviced areas. My Department and the HSE are currently undertaking a Strategic Review of General Practice. The review, with input from key stakeholders, will examine issues affecting general practice including issues around GP capacity. Engagement with relevant stakeholders, including the organisation the Deputy referred to, is ongoing on the thematic issues of the review such as the possible use of salaried GPs and others ways to bring GPs to underserved areas. Furthermore, Primary Care Centres (PCCs) continue to be utilised to serve local communities across the country. As of Q4 last year 174 PCCs were operational including 145 PCCs located outside of Dublin. Seven further PCCs are in construction and due to be operational this year. Finally, it is worth stating that annual investment of €195m was secured under Budget 2024 to enable the continued expansion of primary care services through the Enhanced Community Care (ECC) programme. Amongst other initiatives, the funding has facilitated the rollout of 96 Community Healthcare Networks (CHNs) which are now operational nationwide, and is enabling the continued development of 60 Community Specialist Teams for Older Persons and the Management of Chronic Disease. |
| Deputy Michael Ring asked the Minister for Health in view of the extreme pressures that GPs are under at the moment, what further services pharmacists can be allowed to provide to alleviate the pressure on GP practices; and if he will make a statement on the matter. | 0.7537557076766495 | I thank the deputy for their question. GPs are an important part of primary care, and the Department has taken a number of steps to improve access and care provided by GP services. Training for GPs has expanded in the last number of years. The number of GP training places will increase to 350 in July 2024 which will see a total of 1,300 trainees undertaking the programme. This capacity increase is necessary to maintain the health and well-being of our population. Community pharmacists are another crucial part of our healthcare service, and they carry out a number of essential services such as the safe supply of prescription medication, safe supply of medicines and treatments that you don't need a prescription for, advice on common conditions, help with taking medicines correctly, supply of emergency contraception, blood pressure check-ups and vaccinations. Pharmacists are qualified healthcare professionals, and their role can be expanded to alleviate the pressure on GP practices. On 24th July 2023, I established an Expert Task force to support the expansion of the role of pharmacists in Ireland. The remit of the Task force is to identify and support the delivery of specific objectives, which will serve to align services and practices that can be delivered by pharmacists, and pharmacies, with the needs of the health service and patient. I accepted the first recommendation from the Task force at the end of October. The new measure, which will take effect from 1 March 2024, will allow prescribers to write prescriptions with a maximum validity of 12 months if they so wish. Work is ongoing within the Department to finalise the steps needed to ensure that this is implemented into practice for prescribers and pharmacists. The Expert Task force for Pharmacy are currently working on Phase 2 of their terms of reference - Empowering Pharmacists to Prescribe within their scope of Practice. It is expected they will make further recommendations in the coming months. Any recommendations the Task force make will aim to improve access for patients in a safe, person-centred way. |
| Deputy James O'Connor asked the Minister for Health the current system in place by his Department to provide healthcare for residents of IPAS centres and other refugee accommodation; if his Department is providing additional resources to support GP capacity in areas with higher than average numbers of IPAS centres; and if he will make a statement on the matter. | 0.7527800096168205 | As the question relates to service matters, I have requested the Health Service Executive respond directly to the Deputy. |
As we can see, we do get some relevant information, but we also get some unhelpful information (like the first answer).
Also, depending on how specific our question is, this is potentially too much information, and could potentially confuse or mislead the LLM later on.
To help improve this, let’s first take a look at some common, simple metrics that are used to measure retrieval
4.2.1 Retrieval Metrics Overview
4.2.1.1 Recall
Recall measures how much of the ground truth statement is captured through the retrieval. It does so by counting how many tokens in the retrieved context are also present in the ground truth statement.
Higher recall means that more of the ground truth statement is captured in the context.
4.2.1.2 Precision
Precision is like the inverse of recall, it counts how many tokens in the ground-truth statement are present in the retrieved context.
Higher precision means that there is less ‘additional’ information captured in the context.
4.2.1.3 Intersection over Union (IoU)
This is a metric that captures both recall and precision together. It is calculated by dividing the length of the token overlap by the length of token union of the retrieved context and the ground truth.
For example, if we retrieve 200 tokens, the ground truth is 100 tokens, and the overlap between both is 70 tokens, then the IoU would be 70/(200+100-70) = 0.304
4.2.2 Splitting the Documents
When thinking about recall/precision, a natural question that we might ask relates to the size of the documents that are stored in the database. For example, if we are always retrieving the 5 most similar docs, but each of those are 1000 words long, then we are always passing the model 5000 words. This will probably increase the chance of the target information being present, but it will also increase the amount of superfluous information passed to the LLM (i.e., decrease precision)
Therefore, it might be useful to try splitting the documents into chunks of various sizes and see which size works best.
Below is a simple function that does this for us.
(defn split-document [documents chunk-size]
(let [sentences (->> (str/split documents tokenizer/sentence-splitter-re)
(remove #(= (str/trim %) "")))
chunks (->> (partition-all chunk-size sentences)
(mapv #(str/join #" " %)))]
chunks))4.2.3 Calculating Metrics for Retrieval Strategies
To test our retrieval strategies, we will split up the information contained in the dataset ‘answers’ in the following ways:
Chunks of 3 sentences (Chunks 3)
Chunks of 5 sentences (Chunks 5)
Chunks of 10 sentences (Chunks 10)
Chunks of 15 sentences (Chunks 15)
Original answers, i.e., no splitting (Full Docs)
We will also test our original ‘Question Retrieval’ method that we created in the last section (Question Method).
(def embedding-model (AllMiniLmL6V2EmbeddingModel/new))(defn chunked-docs [docs chunking-size]
(->> (mapv #(split-document % chunking-size) docs) ;; In this case chunk the documents individually, because we know that are all separate/discrete answers
(remove empty?)
(reduce into)))To generate metrics, we first retrieve similar documents based on a given question, then we check those documents against the highlighted (answer) text.
Both functions below are mostly identical, except the ‘question method’ function adds a step from looking up an answer based on a retrieved similar question, it also re-used the previously created db-store-questions, so it’s much quicker to run.
(defn calculate-metrics [questions answers chunked-docs & label]
(let [db-store (InMemoryEmbeddingStore/new)
num (count (mapv #(add-doc-to-store % db-store) chunked-docs))
_ (println num)]
(loop [idx (dec (count questions))
results []]
(if (< idx 0)
results
(let [q-embedding (->> (TextSegment/from (nth questions idx))
(. embedding-model embed)
(.content))
matches (->> (. db-store findRelevant q-embedding 5)
(mapv #(.text (.embedded %)))
(str/join " "))]
(recur (dec idx)
(conj results (tokenizer/calculate-retrieval-metrics
(nth answers idx)
matches
:word
(first label)))))))))(defn calculate-metrics-question-retrieval-method [hl-answers hl-questions]
(loop [idx 0
res []]
(if (= idx (count hl-answers))
res
(let [q-embedding (->> (TextSegment/from (nth hl-questions idx))
(. embedding-model embed)
(.content))
q-matches (->> (. db-store-questions findRelevant q-embedding 5)
(mapv #(.text (.embedded %))))
corresponding-ans (->> (tc/select-rows ds #(some #{(:question %)} q-matches))
:answer
(str/join " "))]
(recur (inc idx)
(conj res (tokenizer/calculate-retrieval-metrics
(nth hl-answers idx)
corresponding-ans
:word
"Question Method")))))))Running these functions over several types of chunking strategies takes a while, so we are going to run them and save the results to a file.
(comment
(defonce metric-comparisons
(let [questions highlights-questions
answers highlights-answers
docs (-> ds
(tc/drop-missing :answer)
(tc/drop-rows #(re-find #"details supplied" (% :question))) ;; getting rid of a few extra unhelpful questions/answers
(tc/drop-rows #(re-find #"As this is a service matter" (% :answer)))
:answer)
full-docs-benchmark (calculate-metrics questions answers docs "Full Docs")]
(loop [[x & xs] [3 5 10 15]
result []]
(if-not x
(conj result full-docs-benchmark)
(let [chunked-docs (chunked-docs docs x)]
(recur xs
(conj result (calculate-metrics questions answers chunked-docs (str "Chunks-" x)))))))))
(spit "data/retrieval_metrics/results.edn"
(with-out-str (clojure.pprint/pprint
(into (reduce into metric-comparisons)
(calculate-metrics-question-retrieval-method highlights-answers highlights-questions))))))(def comparison-data (edn/read-string (slurp "data/retrieval_metrics/results.edn")))(kind/table comparison-data)| recall | precision | IoU | label |
|---|---|---|---|
| 1.0 | 0.04568528 | 0.02970297 | Chunks-3 |
| 0.41666666 | 0.042735044 | 0.01923077 | Chunks-3 |
| 1.0 | 0.125 | 0.06465517 | Chunks-3 |
| 1.0 | 0.09803922 | 0.065789476 | Chunks-3 |
| 0.72727275 | 0.07619048 | 0.034632035 | Chunks-3 |
| 1.0 | 0.043269232 | 0.028571429 | Chunks-5 |
| 0.41666666 | 0.029411765 | 0.01305483 | Chunks-5 |
| 1.0 | 0.07692308 | 0.042134833 | Chunks-5 |
| 1.0 | 0.060728744 | 0.038071066 | Chunks-5 |
| 0.8181818 | 0.05263158 | 0.026946107 | Chunks-5 |
| 0.33333334 | 0.009063444 | 0.005444646 | Chunks-10 |
| 1.0 | 0.039087947 | 0.01775148 | Chunks-10 |
| 1.0 | 0.05033557 | 0.023364486 | Chunks-10 |
| 1.0 | 0.040983606 | 0.022421524 | Chunks-10 |
| 0.72727275 | 0.028268551 | 0.011661808 | Chunks-10 |
| 0.33333334 | 0.0074626864 | 0.004497751 | Chunks-15 |
| 1.0 | 0.032967035 | 0.011684518 | Chunks-15 |
| 1.0 | 0.02970297 | 0.014464802 | Chunks-15 |
| 1.0 | 0.036585364 | 0.019455252 | Chunks-15 |
| 1.0 | 0.03405573 | 0.013546798 | Chunks-15 |
| 1.0 | 0.023622047 | 0.013867488 | Full Docs |
| 1.0 | 0.028436018 | 0.00949367 | Full Docs |
| 1.0 | 0.025996534 | 0.012427506 | Full Docs |
| 1.0 | 0.036585364 | 0.019455252 | Full Docs |
| 1.0 | 0.025114154 | 0.0102325585 | Full Docs |
| 1.0 | 0.027227722 | 0.012061403 | Question Method |
| 1.0 | 0.028571429 | 0.015576324 | Question Method |
| 1.0 | 0.053380784 | 0.035885166 | Question Method |
| 1.0 | 0.02970297 | 0.014760148 | Question Method |
| 1.0 | 0.028213166 | 0.013782542 | Question Method |
(defn average [coll]
(float
(/ (apply + coll)
(count coll))))(def ds-metrics-avg
(->
(tc/dataset comparison-data)
(tc/group-by [:label])
(tc/aggregate {:avg-recall #(average (% :recall))
:avg-precision #(average (% :precision))
:avg-IoU #(average (% :IoU))})))(-> ds-metrics-avg
(plotly/layer-line
{:=x :label
:=y :avg-recall}))(-> ds-metrics-avg
(plotly/layer-line
{:=x :label
:=y :avg-precision}))(-> ds-metrics-avg
(plotly/layer-line
{:=x :label
:=y :avg-IoU}))The results are mostly as expected - with larger chunk sizes recall goes up, although it notably dips with chunk sizes of 10. At the same time, precision goes down - there is more excess information that is perhaps not needed.
Interestingly, the ‘question retrieval’ method has slightly higher precision than the method of looking up the full docs, even though in both cases what is returned are full answers. This suggests that perhaps the ‘naive’ approach of searching for similar questions and returning the corresponding answers is perhaps slightly more precise that searching through the full answers.
Still, it looks like using either chunks of 3 or 5 sentences might be the best overall. For the RAG application, we’ll try using chunks of 3 sentences.
Let’s build that database and save it to a file.
(comment
(let [answers (-> ds
(tc/drop-missing :answer)
(tc/drop-rows #(re-find #"details supplied" (% :question)))
(tc/drop-rows #(re-find #"As this is a service matter" (% :answer)))
:answer)
docs (-> (chunked-docs answers 3)
distinct) ;; before filtering for duplicates there were around 24K chunks, after filtering around 18K
db-store (InMemoryEmbeddingStore/new)
_c (count (mapv #(add-doc-to-store % db-store) docs))]
(println _c)
(spit "data/retrieval_store/db-store-docs.json" (.serializeToJson db-store))))(def db-store-chunked-answers (InMemoryEmbeddingStore/fromFile "data/retrieval_store/db-store-docs.json"))4.2.4 Final Checks
Let’s have a look at the context that is actually generated by each of the approaches to see the difference that the alternate retrieval strategy can make.
(defn gererate-context-question-retrieval [question]
(let [related-questions (->> (query-db-store question 5)
(mapv :text))
past-answers (-> ds
(tc/select-rows #(some #{(:question %)} related-questions))
:answer)]
(mapv (fn [a] {:text a}) past-answers)))(defn generate-context [question db-store-name]
(if (= db-store-name :question-retrieval)
(gererate-context-question-retrieval question)
(let [emb-question (.content (. embedding-model embed question))
related-docs (. db-store-chunked-answers findRelevant emb-question 5)]
(map (fn [doc]
{:text (.text (.embedded doc))
:score (.score doc)})
related-docs))))First we’ll look at a very general question about GP services:
(kind/table
(generate-context "What is the government doing to help improve GP services?" :db-docs))| text | score |
|---|---|
| Following it's completion a paper is to be presented to me outlining the findings of the review and setting out recommended actions to respond to the identified findings. It is worth noting that the Government has undertaken measures to increase GP capacity and improved access to services. Increased annual investment of over €211 million was provided for under the 2019 GP Agreement, providing for increased GP fees, supports and new services. | 0.8480275472660578 |
| The Government is aware of the workforce issues currently facing general practice, and that these challenges limit access to GP services in certain areas and can impact on the timely provision of care. While some issues are specific to certain areas, there is undoubtedly a general increase in demand for GP services due in part to demographics and our aging population. The Government has implemented a range of measures in recent years to increase the number of GPs practicing in the State so that patients across the country have access to GP services. | 0.8396318478023013 |
| The review, with input from key stakeholders, will examine the broad range of issues affecting general practice, including issues related to GP capacity, and will set out the measures necessary to deliver a more sustainable general practice into the future. The provision of GP services in certain rural and urban deprived areas that are underserved, and the possible mechanisms to attract more GPs to these locations, is a specific issue that will be examined under the review. | 0.803669735176046 |
| The programme will help to quickly improve access to GP services, particularly in areas with limited access. Lastly, my Department and the HSE have commenced a Strategic Review of General Practice. The review, with input from key stakeholders, is examining the broad range of issues affecting general practice, including issues related to GP capacity, and when completed will set out the measures necessary to deliver a more sustainable general practice into the future. | 0.7931240722359816 |
| I thank the deputy for their question. GPs are an important part of primary care, and the Department has taken a number of steps to improve access and care provided by GP services. Training for GPs has expanded in the last number of years. | 0.7920156651052908 |
These answers are not a bad starting point for answering this kind of broad question.
Looking at the first answer in the table above, the figure of ‘211m EUR’ is referenced in relation to the 2019 GP Agreement. Let’s see if the database can match this exact figure:
(kind/table
(generate-context "How much annual investment was provided under the 2019 GP agreement?" :db-docs))| text | score |
|---|---|
| Following it's completion a paper is to be presented to me outlining the findings of the review and setting out recommended actions to respond to the identified findings. It is worth noting that the Government has undertaken measures to increase GP capacity and improved access to services. Increased annual investment of over €211 million was provided for under the 2019 GP Agreement, providing for increased GP fees, supports and new services. | 0.8447438929326309 |
| The 2019 GP Agreement provided for increased annual investment of over €211 million for general practice, providing for increased GMS GP fees, supports and new services. In particular the 2019 Agreement increased the enhanced supports package for rural GP practices by 10%. A further €30 million has been provided for under the GP Agreement 2023 for practice supports to expand and retain practice staff. | 0.8420421618721008 |
| The 2019 GP Agreement provided for increased annual investment of over €211 million for general practice, providing for increased GMS GP fees, supports and new services. A further €30 million has been provided for under the GP Agreement 2023 for practice supports to expand and retain practice staff. The annual intake of doctors into the GP training programme has been increased by 80% from 2015 to 2023; there were 286 new entrants to GP training in 2023 and it is planned to increase the number of places for new entrants to 350 for this year. | 0.836534283302552 |
| The Government is aware of the workforce issues currently facing general practice, including the limited access to GP services in certain areas, and is working to ensure that general practice is sustainable in all areas into the future. Under the 2019 GP Agreement additional annual expenditure provided for general practice has been increased now by €211.6m. This provides for significant increases in capitation fees for participating GMS GPs, and new fees and subsidies for additional services. | 0.7993418255841742 |
| Under the 2019 GP Agreement additional annual expenditure provided for general practice was increased by €211.6m. This provided for significant increases in capitation fees for participating GMS GPs, and new fees for additional services and increased practice supports. The GP Agreement 2023 further increased GP capitation fees, increased the existing subsidy rates for practice staff, and introduced a grant support for additional staff capacity as well a practice staff maternity leave support. | 0.7964115032803492 |
Every document retrieved seems to contain the relevant figure :)
For completness, let’s try this same, more specific, question with the previous approach. As you can see below It’s much less focused!
(kind/table
(generate-context "How much annual investment was provided under the 2019 GP agreement" :question-retrieval))| text |
|---|
| The Warmer Homes Scheme delivers free energy upgrades for eligible homeowners in lower income households who are most at risk of energy poverty. There are a number of home energy improvements offered as part of the Scheme. The upgrades that will be recommended for a property will depend on many factors, including age, type and condition of the property. Preliminary data from SEAI shows that last year, a total of approximately 5,900 homes were upgraded under the scheme with an average spend per home of €24,000. As home types vary for each eligible home, SEAI’s technical surveyor will determine which upgrades can be installed and funded. The survey and works recommended are informed by the Building Regulations, incorporating NSAI S.R.54: 2014 Code of Practice for the energy efficient retrofit of dwellings. With specific regard to pre-1940s stone-built dwellings, all eligible homes are assessed and where they comprise stone, SEAI will not insulate the stone walls. These walls are not suitable for the NSAI approved products and standards for retrofitting which are used on more modern construction types. If these walls are insulated using materials or techniques suitable for modern construction, vapour can become trapped leading to problems which can cause harm to the building and its occupants. However, SEAI will assess applicant homes which are stone built for other measures apart from wall insulation which may be possible. If a home is suitable for attic insulation this can be provided. SEAI will also assess, and insulate where appropriate, any post-1940 extensions that may be present in the same home. There can be instances however where the application will be cancelled following this assessment as it may not be possible for any appropriate works to be recommended. These reasons might be that no further works are possible e.g. attic and wall insulation is already there, or that the house is not technically possible to insulate. To support the appropriate upgrade of traditional buildings and stone built homes, a new Guidance Document - “Energy Efficiency in Traditional Buildings” has recently been published by the Department of Housing, Local Government & Heritage. This guidance will be an important resource to SEAI as well as the retrofit and wider construction industry in the appropriate energy efficiency upgrading of older, traditionally built buildings while ensuring compliance with the appropriate building regulations. My officials have asked SEAI to consider the implications of the new Guidance for the operation of the residential and community energy upgrade schemes. There are also other Government grants and schemes available that could be of assistance where there is an urgent need for home repairs. These include the Department of Social Protection Urgent Needs Payment available through the Community Welfare Service at local DSP offices, and the Housing Adaptation Grant Scheme which is administered by the Local Authorities. SEAI has established a specific email address for queries from Oireachtas members so that such queries can be addressed promptly, in line with SEAI’s objective to deliver services to the highest standards. The email address is |
| I propose to take Questions Nos. 323 and 368 together.My Department and Revenue have, for some time, been aware of issues arising from contractual arrangements within the General Practitioner (GP) community whereby some GPs treat income under their General Medical Services (GMS) contract as income of a GP practice in which they are a partner or an employee, rather than income of that individual GP. Revenue issued a guidance note to tax practitioners through the Tax Administration Liaison Committee in July 2023 clarifying the correct tax treatment of GMS income under tax legislation. That guidance confirmed there would be a transitional period for compliance with existing tax law, to 31 December 2023. Supplementary guidance on this matter was published on 10 November 2023 on the Revenue website. In accordance with Section 58 of the Health Act 1970, a GMS contract is between the HSE and an individual GP. This means that, as a matter of law, income under a GMS contract belongs to the GP who entered into the contract with the HSE. The position does not change because a GP treats their GMS income as income of a medical practice. Following on from that fact, the legislation at the time did not provide for a legal basis for Revenue to treat income arising under a GMS contract entered into between an individual GP and the HSE as if it were income arising under a contract between the HSE and the medical practice in which the GP is a partner or an employee. A GP who holds a GMS contract is a chargeable person as regards income arising under that contract and should report such income under the self-assessment system. The GP is also the specified person for the purposes of Professional Services Withholding Tax (PSWT), which means they are entitled to claim a credit for PSWT deducted by the HSE on GMS payments. Although the guidance issued by Revenue was widely reported as a proposed tax change, it did not, in fact, introduce a change to the tax treatment of GPs. Instead, it simply clarified the existing legal and administrative position. However in an effort to find a solution to this issue, discussions took place between officials in the Department of Finance, Revenue, the HSE and the Department of Health. As part of Finance (No. 2) Act 2023, I introduced a new section 1008A into Part 43 of the Taxes Consolidation Act 1997. The section provides that, where individual GPs enter into contracts with the HSE to provide certain medical professional services and provide those services in the conduct of a partnership profession with other individual GPs, the income from those professional services can be treated for income tax purposes, to be that of the partnership. PSWT credits may be claimed by the partnership under such instances. Where this occurs, the partner who has the contract with the HSE, and not the precedent partner of the medical partnership, should provide the tax number of the medical partnership concerned to the HSE for the purposes of PSWT. This provision will ensure that, where elected, all amounts paid to, or for the benefit of, a GP by the HSE in respect of GMS and ancillary public services can be treated as income of the partnership. A joint election to treat the GMS and ancillary public services as income of the partnership must be made by the GP contracted by the HSE to provide the relevant medical services and the medical partnership concerned. Revenue guidance has been updated to take account of this, with further information available at: www.revenue.ie/en/tax-professionals/tdm/income-tax-capital-gains-tax-corporation-tax/part-04/04-01-15.pdf. The provision will only apply to income tax for individual doctors who operate in partnerships with other individual doctors and will not apply to, or change, the tax situation for doctors who are employees of a partnership or other arrangement, including corporates. It is also limited only to income arising from General Medical Services (GMS) and certain ancillary public income. It would not be appropriate for tax legislation to seek to accommodate all contracts and business practices in the sector. The core issue concerns the contractual arrangements involving GPs, which is a matter under the remit of the Minister for Health. As such, the Minister for Health has confirmed that the Strategic Review of General Practice, which is now underway, with input from key stakeholders, will examine the broad range of issues affecting general practice, including the relevant HSE contracts and propose measures necessary to modernise them. |
| I propose to take Questions Nos. 323 and 368 together.My Department and Revenue have, for some time, been aware of issues arising from contractual arrangements within the General Practitioner (GP) community whereby some GPs treat income under their General Medical Services (GMS) contract as income of a GP practice in which they are a partner or an employee, rather than income of that individual GP. Revenue issued a guidance note to tax practitioners through the Tax Administration Liaison Committee in July 2023 clarifying the correct tax treatment of GMS income under tax legislation. That guidance confirmed there would be a transitional period for compliance with existing tax law, to 31 December 2023. Supplementary guidance on this matter was published on 10 November 2023 on the Revenue website. In accordance with Section 58 of the Health Act 1970, a GMS contract is between the HSE and an individual GP. This means that, as a matter of law, income under a GMS contract belongs to the GP who entered into the contract with the HSE. The position does not change because a GP treats their GMS income as income of a medical practice. Following on from that fact, the legislation at the time did not provide for a legal basis for Revenue to treat income arising under a GMS contract entered into between an individual GP and the HSE as if it were income arising under a contract between the HSE and the medical practice in which the GP is a partner or an employee. A GP who holds a GMS contract is a chargeable person as regards income arising under that contract and should report such income under the self-assessment system. The GP is also the specified person for the purposes of Professional Services Withholding Tax (PSWT), which means they are entitled to claim a credit for PSWT deducted by the HSE on GMS payments. Although the guidance issued by Revenue was widely reported as a proposed tax change, it did not, in fact, introduce a change to the tax treatment of GPs. Instead, it simply clarified the existing legal and administrative position. However in an effort to find a solution to this issue, discussions took place between officials in the Department of Finance, Revenue, the HSE and the Department of Health. As part of Finance (No. 2) Act 2023, I introduced a new section 1008A into Part 43 of the Taxes Consolidation Act 1997. The section provides that, where individual GPs enter into contracts with the HSE to provide certain medical professional services and provide those services in the conduct of a partnership profession with other individual GPs, the income from those professional services can be treated for income tax purposes, to be that of the partnership. PSWT credits may be claimed by the partnership under such instances. Where this occurs, the partner who has the contract with the HSE, and not the precedent partner of the medical partnership, should provide the tax number of the medical partnership concerned to the HSE for the purposes of PSWT. This provision will ensure that, where elected, all amounts paid to, or for the benefit of, a GP by the HSE in respect of GMS and ancillary public services can be treated as income of the partnership. A joint election to treat the GMS and ancillary public services as income of the partnership must be made by the GP contracted by the HSE to provide the relevant medical services and the medical partnership concerned. Revenue guidance has been updated to take account of this, with further information available at: www.revenue.ie/en/tax-professionals/tdm/income-tax-capital-gains-tax-corporation-tax/part-04/04-01-15.pdf. The provision will only apply to income tax for individual doctors who operate in partnerships with other individual doctors and will not apply to, or change, the tax situation for doctors who are employees of a partnership or other arrangement, including corporates. It is also limited only to income arising from General Medical Services (GMS) and certain ancillary public income. It would not be appropriate for tax legislation to seek to accommodate all contracts and business practices in the sector. The core issue concerns the contractual arrangements involving GPs, which is a matter under the remit of the Minister for Health. As such, the Minister for Health has confirmed that the Strategic Review of General Practice, which is now underway, with input from key stakeholders, will examine the broad range of issues affecting general practice, including the relevant HSE contracts and propose measures necessary to modernise them. |
| To increase the number of GPs working in the country and thereby improve access to GP services, the annual intake of doctors into the GP training programme has been increased by over 80% since 2015. Last year, there were 286 new entrants to GP training, and 350 places for new entrants are planned for this year. The ICGP, which is responsible for training GPs, indicated that 968 applications were received for 2023, while 1,311 applications were received for this year. Budget 2024 provides for the increase to 350 places for new entrants to GP training for 2024. The annual year one cost of a doctor on the GP training programme is approximately €105,000, accordingly the cost increasing the number of GP training places by a further 25 places to 375 in 2024 would be approximately €2.6 million. However, it is important to note that capacity to increase the number of GP trainees must be built through the health service, as for the first two years of training GP trainees rotate through a variety of hospital specialities. Expanding the number of such places available for GP trainees may also carry additional costs which cannot be quantified at this time. |
source: src/notebooks/vdb_evaluation.clj