by Pavel A. Mozolyako and Nikolai N. Osipov
Probability estimates for different prognoses in the medical field may be achieved by means of a global system of weighted expert assessments. The system, based on the concept of a virtual prediction market, will allow aggregation of the intuitive opinions of various experts about outcomes of a medical case.
Imagine a process with several possible outcomes that are mutually exclusive: for example, an illness that may result in either recovery or death. A pool of experts is available to offer their intuitive opinions about the likely outcome. The following question naturally arises: how do we aggregate these opinions to obtain probability estimates for outcomes? The answer: by using a virtual prediction market.
How to transform the intuition of many experts into one probability estimate? (Picture source: http://consiliummanagement.com/investment-management/).
Commercial prediction markets are systems where people bet with each other on possible outcomes of an event, choosing various odds (prices) and amounts of money to risk. Equilibrium prices in such a market are known to give good probability estimates for outcomes [1]. However, if such a market is commercial, this activity is close in spirit to bookmaking and cannot be considered ethical (especially when applied to medicine). We can, however, organize a virtual analogue of such a market, using virtual points (‘votes’) instead of money, to create an excellent system of weighted voting. An ‘expert’ is anyone with an opinion about the process and its outcomes; unskilled ‘experts’ have little impact on the process since their collective weight in the vote pool is small. In this sense, prediction markets are very stable systems.
At least one such system is already successfully applied in medicine. This is the CrowdMed project, which is designed to provide sophisticated diagnoses by means of weighted voting of a large number of experts. For example, it allows diagnosis of nontrivial genetic abnormalities. This system also allows a solution to be selected, but without a detailed analysis of its possible effects.
The system we are designing will be mainly intended to assist with choice of treatment for already diagnosed patients. For each case, the system will analyse in detail all possible effects for each solution that is proposed either by the patient's attending physician, by the patient, or by another expert participating in the voting. So while CrowdMed is mainly dedicated to making sophisticated diagnosis, our system is intended for cases where we have nontrivial solutions with effects that are difficult to forecast.
A patient with a confirmed diagnosis may have several variants of treatment to choose from (for example, ‘no treatment’, ‘surgical treatment’, ‘drug treatment 1’, and ‘drug treatment 2’). For each variant, the attending doctor may describe one or more possible effects, and will open voting (implemented as a virtual prediction market) for each of them. For example, the doctor could add ‘the patient will survive for five years’ effect for the ‘no treatment’ variant; ‘the patient will survive the operation’ and ‘the patient will survive for five years’ effects for the ‘surgical treatment’ variant, and so on. After the voting, we will obtain some estimates for the corresponding conditional probabilities (for example, the conditional probability of the event ‘the patient will survive for five years’ given the event ‘surgical treatment’), and the doctor will be able to choose the most appropriate treatment.
How should the above estimates be calculated? As we have said before, some probability estimates are given by equilibrium prices of the corresponding virtual markets. But such estimates are too rough, and a lot of time is needed to achieve the equilibrium state. We are developing a method that will allow effective utilization of prediction market data and extraction of an aggregated opinion of experts. Our approach is based on one of the latest concepts of decision theory (lottery dependent utility) [2], analysis of censored samples, and some equilibrium equations. We also aim to compare the methods of classical medical statistics with our approach. Namely, we are going to show that our system gives probability estimates that are at least as accurate as those obtained by the classical method of regressions on medical data [3].
This project has been running since 2014. It is a joint project between Alexandra Yu. Kalinichenko (SPb. Inform. and Analyt. Center), Dina Yu. Kalinichenko (SPbSU), Pavel A. Mozolyako (NTNU), Alexey V. Osipov (OLMA invest. comp.), Nikolai N. Osipov (NTNU and PDMI), and Dmitry V. Ponomarev (Inria).
Link:
CrowdMed project: http://www.crowdmed.com/
References:
[1] Lionel Page and Robert T. Clemen, Do prediction markets produce well-calibrated probability forecasts?, The Econom. J., Vol. 123, No. 568, 491–513, 2013
[2] Michèle Cohen, Security level, potential level, expected utility: a three-criteria decision model under risk, Theory and Decision, Vol. 33, No. 2, 101–134, 1992
[3] James K. Lindsey, Applying Generalized Linear Models, Springer Texts in Statistics, 2000
Please contact:
Nikolai N. Osipov
ERCIM research fellow, NTNU, Norway
E-mail:
