Research has found that AI is completely defeated by human!
Writer： admin Time：2024-07-30 16:08 Browse：℃
Research has found that AI is completely defeated by human!
Will human doctors be laid off due to big models such as ChatGPT?
This concern is not groundless. After all, Google's big model (Med PalM 2) has easily won the US medical licensing exam and reached the level of a medical expert.
However, a recent study suggests that in clinical practice, human doctors outperform current artificial intelligence (AI) models without worrying too much about personal "unemployment issues".
The related research paper titled "Evaluation and mitigation of the limitations of large language models in clinical decision-making" has been recently published in the scientific journal Nature Medicine.
The study found that even the most advanced Large Language Models (LLMs) currently available cannot make accurate diagnoses for all patients and perform significantly worse than human doctors——
The diagnostic accuracy of doctors is 89%, while the diagnostic accuracy of LLM is only 73%. In an extreme case (diagnosis of cholecystitis), the accuracy of LLM is only 13%.
What is even more surprising is that as the understanding of case information increases, the diagnostic accuracy of LLM may actually decrease, and sometimes even require some examinations that may pose serious health risks to patients.
How did LLM perform as an emergency department doctor?
Although LLM can easily pass the US medical licensing exam, the medical licensing exam and clinical case challenges are suitable for testing only the general medical knowledge of candidates, and the difficulty is far less than the daily complex clinical decision-making tasks.
Clinical decision-making is a multi-step process that requires collecting and integrating data from different sources, and continuously evaluating facts to achieve evidence-based patient diagnosis and treatment decisions.
In order to further investigate the potential of LLM in clinical diagnosis, a research team from the Technical University of Munich and its collaborators created a dataset covering 2400 real patient cases and four common abdominal diseases (appendicitis, pancreatitis, cholecystitis, and diverticulitis) based on the Medical Information Market Intensive Care Database (MIMIC-IV), simulating the real clinical environment and reproducing the process from emergency to treatment, in order to evaluate its applicability as a clinical decision maker.
Dataset source and evaluation framework. This dataset is sourced from real cases in the MIMIC-IV database and includes comprehensive electronic health record data recorded during hospitalization. The evaluation framework reflects a realistic clinical environment and comprehensively evaluates LLM from multiple criteria, including diagnostic accuracy, adherence to diagnostic and treatment guidelines, consistency in following instructions, ability to interpret laboratory results, and robustness to changes in instructions, information volume, and information sequence. ICD， International Classification of Diseases; CT， Computed tomography; US， ultrasonic; MRCP， Magnetic resonance cholangiopancreatography.
The research team tested Llama 2 and its derivative versions, including generic versions such as Llama 2 Chat, Open Assistant, WizardLM, and medical domain aligned models such as Clinical Camel and Meditron.
Due to privacy concerns and data usage protocols of MIMIC, the data cannot be used for external APIs such as OpenAI or Google, therefore ChatGPT, GPT-4, and Med PalM could not be tested. It is worth noting that Llama 2, Clinical Camel, and Meditron have achieved or even surpassed ChatGPT in medical licensing exams and biomedical Q&A tests.
The control group consisted of four internal medicine physicians from two countries with different years of emergency experience (2, 3, 4, and 29 years, respectively). The results showed that LLM's performance in clinical diagnosis is far inferior to that of human doctors.
In fact, as early as the 2008 financial crisis, the Indian software industry had a software outsourcing company that exploded. This Indian software company was called Satiyang, which developed earlier and was already a giant in the Indian outsourcing software industry in the late 1990s. It went public in New York in 2001 and saw a 48% increase in revenue and a 36% increase in profits in the fiscal year 2007-2008. During the financial crisis, its financial data fraud was exposed, with over 90% of its performance being fabricated.
But before that, American technology Internet companies also supervised software, so there were far fewer quality problems. Indian outsourcing companies also had to be more careful when doing software. However, with many Indians becoming CEOs, the software quality of American technology Internet can be seen from today's Boeing and Microsoft.
Previously, the industry had analyzed that most of the American technology Internet was headed by Indians, while most Chinese engineers could only work as code farmers.
Industry insiders believed that this was because Indians and American technology Internet bosses were easier to communicate with each other. On the other hand, Indians were happy to have a group of Indians. When one person rose to the top, they pulled a bunch of Indians. Everyone gathered firewood and finally succeeded in pushing Indians to the position of CEO of the American technology Internet. Chinese engineers preferred to work hard, which can be seen from the fact that the chip industry, which is more technically rigorous, is generally headed by Chinese, and Chinese engineers are responsible for technology development.
Boeing and Microsoft have had problems since they became CEO of Indian origin, which may make the American technology Internet industry think about the advantages and disadvantages of having an Indian origin as CEO. Boeing has quickly dismissed the CEO of Indian origin, and the new generation of American millionaire Mask is not interested in Indian origin. Mask fired the CEO of Indian origin at the first time after he bought Twitter. Maybe Indian origin will be left out in the American technology Internet industry from now on.
On July 19th, Microsoft Blue Screen appeared on the hot search list. The pictures shared by multiple users on social media showed that the company's computer had a blue screen and prompted "The device is experiencing a problem and needs to be restarted." As a result, many users left work early on Friday afternoon.
This is a global IT system failure. Multiple countries and regions' flight systems, banking systems, and supermarket systems have been affected and 'paralyzed'. The airport system in Hong Kong, China has also been affected. On the same day, Microsoft officially announced that it is investigating issues affecting users' access to various Microsoft 365 applications and services.
Subsequently, the blue screen issue...
Industry cybersecurity experts analyzed to reporters from Beike Finance that the blue screen malfunction is most likely caused by a program error in the antivirus software, which triggered the self-protection mechanism of the Windows system. Although the company actively remedied the problem after it occurred, it still caused significant losses to users worldwide. In the era of cloud computing, the challenges faced by business system stability will be more severe than ever before. Enterprises urgently need to strengthen their security awareness, respond to technical failures reasonably, prepare emergency plans in advance, and re evaluate their business stability guarantees, disaster recovery plans, and risks of technological dependence.
We QCIT Group of companies beleive that anyone, who is physically eliminating others for any reason always, is the weak person and they are and will not be the right countries' people on the technical jobs. And as the indirect partners of the Microsoft CORP., we did not suffered this, it is not the luck, because we do know at least the fine technology needs the real technicians to run and operate only.
Skype: Qiaochuit