Accident or murder: image recognition helps Shanghai CSIs determine cause of death
In the course of 2018 Labworld from Shanghai knocked onSioux Technologies' door in search of specialized image recognition technology. In doing so, they hoped to improve an analysis process used to determine whether a drowning is an accident or murder. Murder by drowning is relatively common in China due to the absence of firearms. In order to clarify the cause of death in drowning victims, forensic experts use electron microscopes to search for diatoms in samples of lung, kidney or liver tissue. These are single-celled algae found everywhere in natural waters. There are as many as ten thousand different species, all different in shape and size. Due to all kinds of influences, all surface waters have their own specific populations. These can therefore serve as a fingerprint of the crime scene. If the algae in the surface water at the supposed site of the drowning do not match the composition of the water in the lungs, then something is wrong.
The importance of speaking the language of the end customers
Labworld knew Sioux from Phenom-World, now part of Thermo Fisher Scientific. The Chinese company supplies Thermo Fisher's table-top electron microscopes along with its self-developed analysis tools to end customers. These tools are aimed at CSI-like investigations into murders, among other things. Labworld saw opportunities for improvement of the product's image recognition capabilities. Martijn Kabel, innovation manager at Sioux Technologies: "Labworld thought that automatic image recognition would speed things up considerably. That's what we thought at first." Sioux started by mapping out the entire workflow. To do this, an employee of their Chinese branch visited forensic experts in Guangzhou. The provincial police station there is recognised as an authority in the field of forensic investigation based on diatoms. Kabel: "It was very important that our colleague spoke the language and at the same time knew what was possible within Sioux".
The Sioux employee observed experts creating images manually. Then they identified and counted the diatoms. Using this method it took several days to investigate a single drowning case. "Because we mapped the workflow very precisely, we saw that scanning and analyzing samples one by one resulted in too much man-machine interaction. Scaling up was almost impossible. More sophisticated image recognition wouldn't change that much either," says Kabel. It became clear that the real gain was to be found in reducing the scanning time.
Object recognition and not image recognition turned out to be the key to saving time
The workflow analysis showed that the assessment process would speed up enormously if the forensic expert was helped with an initial rough assessment: whether there are any diatoms in the image at all. Sioux thought this could be done automatically. The computer then had to make a pre-selection of the images on which diatoms might be visible. Experts would then be needed for the final assessment: whether candidate diatoms were indeed diatoms and of what type.
For the automatic preselection of diatoms, object recognition algorithms based on Machine Learning technology from the world of autonomous driving proved to be useful. Rob Knoops, a mathematical engineer involved in the Labworld project: "We put a signal square around candidate diatoms in the microscope image, just like the object recognition algorithm in autonomous driving puts a square around potential traffic signs and people. The next step is to turn these candidates into a higher resolution image. The forensic expert can then assess whether it is indeed a diatom or not. With this rough preselection you will get misses, but they are acceptable".
In this way, time was saved in this specific workflow by detecting potential hits at the lowest possible resolution. After that, only higher resolution images were created from potential hits. With this method, less high-resolution images are needed, which reducing the time spent on the electron microscope.
An optimal user interface provides even more improvement
Focus on workflow and value more important than technology
Kabel emphasizes that it is important to focus on the workflow and the value stream itself. Technicians often tend to think from technical possibilities, but often these are not the limiting factors. Rather, these are time and budget. "Therefore, focus on what is important to the customer or your product and try to determine within the time and budget available how you can achieve maximum results by smart data collection."
Knoops also emphasizes the perspective of the end user: "For the forensic expert it is not necessary at all to make a nice picture of a diatom - although this is technically possible and it produces brilliant photos. It's not even important to recognise diatoms perfectly automatically. Experts mainly want to know that it is a diatom and of what kind. The working method allows forensic experts to save a considerable amount of time, which means that drowning cases can now be dealt with much more quickly".