Murder or accident: image recognition helps Shanghai determine cause of death
Important to speak the language of the end customer
Labworld knows Sioux from Phenom-World, now part of Thermo Fisher Scientific. The Chinese company supplies Thermo Fisher's table electron microscopes with its self-developed analysis tools. These are aimed at CSI-like investigations into murders, among other things. Labworld saw opportunities for improvement with image recognition. Martijn Kabel, innovation manager at Sioux Technologies: "Labworld thought that automatic image recognition would speed things up considerably. We thought so too at first." However, Sioux started by mapping out the entire workflow. To do so, an employee of their Chinese branch visited forensic experts in Guangzhou. The provincial police station there is known as an authority in the field of diatom-based forensics. Kabel: "It was very important that our colleague spoke the language and at the same time knew what was possible within Sioux".
The Sioux employee saw experts taking images manually. Then they identified and counted the diatoms. In this way, it takes several days to investigate a single drowning case. "Because we mapped the workflow very precisely, we saw that with one-by-one scanning, and analysis, there was far too much human-machine interaction required. Scaling up was almost impossible. Image recognition would not change that much either," says Kabel. It became clear that the real gain was in reducing the scanning time.
Object recognition and not image recognition turns out to be key to saving time
The workflow analysis revealed that the assessment process would be greatly accelerated if the forensic expert were assisted with an initial rough assessment: whether there were any diatoms visible in the image at all. Sioux thought this could be done automatically. The computer would then have to make a pre-selection of the images on which diatoms might be visible. Experts would then only be needed for the final assessment: whether the images are indeed diatoms and which ones they are.
For the automatic pre-selection of diatoms, object recognition algorithms based on Machine Learning technology from the world of autonomous driving turned out to be useful. Rob Knoops, involved in the Labworld project as a mathematical engineer: "We put a signal square around potential diatoms in the microscope image, just like the object recognition algorithm in autonomous driving puts a square around potential road signs and people. As a next step, you can create a higher resolution image of these potential hits. The forensic expert can then assess whether this is indeed one, or a miss. With this rough pre-selection, you do get misses, but they are acceptable."
In this way, time was saved in this particular workflow by detecting potential hits at the lowest possible resolution. To then take higher resolution images only of potential hits. With this workflow, fewer higher-resolution images are needed, which reduces the time demand on the electron microscope.
Optimal user interface provides another improvement
Focus on workflow and value more important than technology
Kabel emphasizes that it is important to focus on the workflow and the value stream itself. Technicians often tend to think in terms of technical possibilities, but often these are not the limiting factors. Rather, those are time and budget. "Therefore, look primarily at what is important to the customer, or to your product, and try to determine within the time and budget available how you can achieve maximum results through smart data collection."
Knoops also emphasizes the end-user perspective: "It is not at all necessary for the forensic expert to make a nice picture of a diatom - although this is technically possible and it produces wonderful pictures. It is not even important to recognize diatoms perfectly automatically. Experts mainly want to know that it is a diatom and what kind. With the method, forensic experts save considerable time, so drowning cases can now be handled much faster."