Color video technology paired with artificial intelligence holds great development potential for the future optimization of patient care. Mobile systems have an important role to play in this context, and OEMs need solutions that integrate embedded vision and embedded computer technology.

Visual diagnosis is one of the basic medical examination techniques used by practitioners. It involves looking closely at the patient to diagnose visible structural or functional changes and is often performed with the aid of camera systems these days — both externally in dermatology (e.g., to detect skin cancer) and ophthalmology, and internally in endoscopy (e.g., for colonoscopies), as well as in the laboratory in the form of electron microscopy.1

Vision Handhelds for eHealth Applications

Using vision systems to support the human eye has many advantages. With the expansion of the 5G network, it is now also becoming possible to transmit data live and in real time to gather expert opinions. When paired with artificial intelligence, the requirements for such expert opinions can be minimized, because in an adequately trained system, most images can be clearly classified as inconspicuous or conspicuous. While an IT system cannot and should not replace the diagnosis of a medical expert, obtaining an initial assessment or even a second opinion from an AI system could become standard in the future to make patient care better and more efficient.

This fundus camera (photo from 2010) required a camera to take pictures and a PC for analysis. Today’s systems are much more integrated. In the future, AI-based fundus cameras could revolutionize screening for glaucoma, diabetic retinopathy or age-related macular degeneration. (Credit: Wikipedia/Free Art License 1.3)

Applications in Dermatology, Colonoscopy, and Ophthalmology

As scientists at the Massachusetts Institute of Technology (MIT) have discovered, it is possible today to measure a patient’s pulse and respiration using video recordings alone.2 The naked eye cannot recognize minimal changes in face color caused by pulsating blood circulation. There are also already some successful studies for the early detection of skin cancer.3 Scientists from the German cancer research center DKFZ, the Heidelberg University Hospital skin clinic, and the National Center for Tumor Diseases (NCT) in Heidelberg have programmed an algorithm that can digitally assess suspicious skin changes. For 100 images of skin abnormalities, only 7 out of 157 participating dermatologists achieved better results than the algorithm.

Doctors and AI systems alike use images to determine if all the thin vessels of the retina are connected and intact or, when this is not the case, whether follow-up examinations are necessary. This type of diagnosis contributes to the prevention of eye diseases such as macular degeneration.

Research is also currently being conducted into AI-based diagnosis of colorectal cancer. The images acquired during colonoscopy are transmitted to an AI system that uses 300 characteristics to determine whether an anomaly is early stage colorectal cancer or a harmless tumor. Today, common practice is to remove the growth and have it examined in the laboratory — a process that is equally arduous for patients and hospitals.4

Glaucoma, diabetic retinopathy, or age-related macular degeneration (AMD) are other examples of medical conditions that could be detected much earlier if suitable technology was used beyond ophthalmological practices. Such a service could easily be offered by general practitioners as part of annual checkups, or by nursing staff in elderly care homes and home care settings, or even by opticians or pharmacists, similar to blood pressure measurement. Since there is no reliable treatment as yet for AMD, one of the most common eye diseases in the western world causing severe vision loss in people over 50, early detection would help significantly avoid risk factors.5

A Great Future — Mobile Fundus Cameras with AI

Smart embedded vision platforms with AI-based situational awareness consist of many small function blocks whose interaction must be validated. (Credit: Basler AG)

With AI still in its infancy, it will certainly be a while before mobile handheld devices with 5G connectivity and a fully trained AI information system come to the market. Having said that, the technology is ready now for series production in regard to black-and-white x-ray imaging.6 Doubtlessly, color images will sooner or later also be analyzed via AI. However, on the way to an all-in-one mobile retina examination solution with an integrated fundus camera, it will be necessary to develop the vision system from a tabletop device into a highly integrated handheld to make it suitable for mobile applications. This is a big challenge for the manufacturers of such medical diagnostic devices.

The NXP solution platform from Basler and congatec also targets AI applications. Based on a Basler Embedded Vision Kit, it features an NXP i.MX 8QuadMax SoC on the conga-SMX8 SMARC 2.0 Computer-on-Module from congatec, a SMARC 2.0 carrier board, and Basler’s dart BCON camera module for MIPI 13 MP. (Credit: congatec)

Complex Challenges for Fundus Camera Manufacturers. On the one hand, development falls to the solution providers to continuously advance the camera software in general, because such technology continues to be used in stationary applications and needs to be further improved. In addition, 5G connectivity including the cloud as eHealth platform for remote diagnosis, matching server platforms for AI training, and the device itself must all be redeveloped completely. This applies to both the hardware and the software — which constitutes an enormous challenge.

Embedded Camera System and Color Calibration Essential. On the hardware side alone, this often requires manufacturers to weigh decisions that affect the long-term success of the solution. Choosing the right vision system is not easy. In particular, the reliability of the color rendering during image acquisition and reproduction plays a key role in enabling both practitioners and AI systems to assess and classify whether a structure is healthy or diseased. Here, a lot depends on color calibration, which requires that the parameters within the color computation pipeline in the camera firmware are fully optimized. The goal of the calibration process is to parameterize the individual function blocks in the camera’s color pipeline in a way that minimizes color defects ΔE compared to the reference values; avoiding them completely is not possible.

Windows or Linux, ARM or x86?

Then, there’s also the question of which processor architecture to use that needs answering: Is a Linux-based ARM system a better solution, or should the handheld device be designed as an x86 platform and Windows operating system (OS), like previous diagnostic computers? And even when sticking with x86, there is often a decision to be made whether to use an AMD or Intel system. Counting in favor of AMD are SoC-integrated GPGPU processing and the open source philosophy, while the software ecosystem may speak for opting for Intel.

Ultimately, after all these decisions have been taken, there is still the entire inner electronics to be developed, designed into a suitable housing and provided with the appropriate board support package for the OS image, so that the OEM’s application software can do what it is supposed to do: deliver perfect images and a management system that makes diagnosis much more efficient. Since these are complex challenges, embedded vision expert Basler and embedded computing expert congatec, for example, have entered into a partnership to be able to support OEMs throughout their engineering processes, from first requirement engineering to series production, more comprehensively as a one-stop platform provider.

Embedded Vision Competence United

With an interdisciplinary team of experts from both companies taking care of the entire solution platform and validating the interaction between the individual embedded vision and embedded computing components, OEMs need to spend significantly less effort on coordination during development. Customers benefit from accelerated design-in of the “embedded vision computer” component as well as optimized service and support during series production.

Computer-on-modules from congatec are often the technology basis for such projects, because they make it particularly easy to scale the performance and match the closed-loop engineering strategies to the specific requirements. This is an undeniable advantage for the provision of suitable AI systems, as performance requirements are bound to change as the development of AI and inference software progresses. Fusing module, carrier board, and camera into a full-custom OEM solution is another option. Even the development of a completely customized solution platform including housing is part of the joint offer. To make it easier for developers to choose between ARM and x86, SMARC or Qseven computer-on-modules are especially recommended. Available with both architectures, they offer a compact footprint for mobile and handheld devices.


  1. Artificial Intelligence in Healthcare
  2. G. blakrishnan et al., “Detecting Pulse from Head Motions in Video
  3. Andre Esteva et al., “Skin Cancer Classificaton with Deep Learning
  4. Y. Mori, et al., “Real-Time Use of Artificial Intelligence in Identification of Diminutive Polyps During Colonoscopy: A Prospective Study
  5. Macular Degeneration
  6. AIDOC, [in German].

This article was written by Zeljko Loncaric, Marketing Engineer for congatec AG, Deggendorf, Germany, and Felix Chemnitz, Product Market Manager Medical for Basler AG, Ahrensburg, Germany. For more information on congatec AG, visit here . For more information on Basler AG, visit here .