LU103192B1

LU103192B1 - Display system

Info

Publication number: LU103192B1
Application number: LU103192A
Authority: LU
Inventors: Tom Kimpe
Original assignee: Barco Nv
Priority date: 2023-08-31
Filing date: 2023-08-31
Publication date: 2025-02-28
Also published as: CN121941452A; WO2025046117A1

Abstract

The present disclosure relates to a display system which uses a gaze tracking application for recording gaze patten of a user. The display system comprises a display device configures to display an image; a gaze tracking device configured to track gaze direction information indicating where in the image a user is gazing when viewing the image; a processing device configured to generate a gaze pattern using the gaze direction information; and a memory configured to store the gaze pattern. When playing back the gaze pattern, a focused areas on the image are generated according to the gaze pattern to identify areas viewed by the user. In this way, other users will be able to track analysis process of the user.

Description

Title: Display system LU103192

TECHNICAL FIELD

The present disclosure relates to a display system, in particular to a display system having a gaze tracking application allowing interaction between at least two users to exchange precise information.

BACKGROUND

There is a worldwide shortage of medical specialists such as radiologists, pathologists, dermatologists etc. Moreover, diagnosing medical images such as radiology, pathology or dermoscopic images is very complex and requires long training. This creates problems as workload for specialists is increasing and waiting times for patients are getting longer. Moreover, long working hours increases the risk of medical errors. Therefore it is important that residents (radiologists, pathologists, dermatologists in training) can quickly learn how to efficiently, reliable, diagnose medical images with high quality. And there is a need for technological solutions that can help residents, junior radiologists, but also senior radiologists with reading medical imaging quickly and with high clinical performance.

Gaze tracking is the process of measuring and analyzing the movements of a person's eyes to determine where the person is looking. Gaze tracking technology has become mature enough such that larger viewing angles are possible, and the accuracy of the measured data is increasing as well. Recently, it has become possible to integrate gaze tracking systems into a display, while previously gaze trackers typically were separate (camera based) systems that were placed below or next to a display system. Alternatively there are also gaze tracking systems that are wearable and integrated in eg. (smart) glasses.

Gaze tracking technology already has been used in healthcare applications. For example: gaze tracking has been used as alternative human-machine interface where a user can control the mouse pointer and other Ul elements by means of the eyes. By looking at a location or Ul element, the user can control the computer system. Another example is in training context. Gaze tracking has been used in healthcare and other industries to understand how certain complex tasks are being performed by experts. Eg. gaze tracking was used to study how experienced radiologists are reading radiology images. That knowledge then can be used to teach how residents or junior radiologists can improve their skills or efficiency.

2 LU103192

SUMMARY

It is to be understood that both the foregoing general description and the following detailed description are exemplary and are intended to provide further explanation of the invention as claimed. Other advantages and features of the invention will be apparent from the following description, drawings and claims.

According to one aspect of the present disclosure, there is provided a display system. The display system comprising a display device configured to display an image; a gaze tracking device configured to track gaze direction information indicating where in the image a user is gazing when viewing the image; a processing device configured to generate a gaze pattern using the gaze direction information; and a memory configured to store the gaze pattern. When playing back the gaze pattern, one or more focused areas on the image are generated according to the gaze pattern to identify areas viewed by the user.

In some other embodiments, the gaze pattern is played back in another display device of another display system.

In some other embodiments, the display system further comprises a transceiver configured to transmit the gaze pattern to the other display device using a wired or a wireless network.

In some other embodiments, the gaze tracking application is configured to convert the gaze direction information to at least one gaze coordinate of the gaze pattern or gaze path, wherein the at least one gaze coordinate is sent to the other display system through the transceiver to generate a focused area on an image displayed on the other display device.

In some other embodiments, an updated image having the focused area with the image is sent to the other display system for direct viewing of the focused area.

In some other embodiments, when display resolutions of the two display systems are different, the image transmitted is either scaled to match for the two display systems, show only a portion of the image on the display system having lower resolution, or a combination of both, wherein the display system having lower resolution, at least displays a region of interest where the user is focused on.

3 LU103192

In some other embodiments, the gaze tracking application is configured to generate an overview window, the overview window is configured to display a map of the entirety of the image with an indicator of the relative area of the region of interest on the map.

In some other embodiments, a normal speed of playback of the gaze pattern is according to a normal speed of the user going through the image, wherein the speed of the playback is controllable to speed up or slow down.

In some other embodiments, the processing device further comprising a user interface application for controlling the image displayed on the display device, the memory further configured to store additional interactions comprising input to the user interface application when generating the gaze pattern.

In some other embodiments, the gaze tracking application further configured to filter the gaze direction information corresponding to unwanted eye movements to reduce interference in the gaze pattern.

In some other embodiments, the gaze tracking application furthered configured to filter the image to show the focused area, method of filtering comprising blurring, sharpening, (de)focusing, decoloring, enhancing color saturation, contrast enhancement, and/or contrast reduction.

In some other embodiments, the gaze tracking application includes an inputting unit configured to activate or deactivate recording of gaze direction information and/or activate or deactivate playback of the gaze pattern.

In some other embodiments, the gaze tracking application further recognizing an

Al algorithm as the user and configured to record at least one potential area of interest of the

Al algorithm as a focused area for the gaze pattern.

In some other embodiments, the gaze tracking application is further configured to track gaze direction information of another user while playing back the gaze pattern.

In some other embodiments, the gaze tracking application is further configured to record the gaze pattern of the other user for playing back or for comparing with the gaze pattern of the user.

4 LU103192

In some other embodiments, the gaze tracking application is further configured to display a first focused area and only display a second focused area when the gaze tracking application has determined that the other user has focused on the first focused area.

In some other embodiments, the second focused area is displayed after focusing on the first focused area for an expected time period.

In some other embodiments, the gaze tracking application is further configured to display a focused area of the gaze pattern only when the other user has moved focus to a latter focus area of the gaze pattern without focusing on this focused area of the gaze pattern.

In some other embodiments, the gaze tracking application is configured to record the gaze pattern with corresponding time stamps.

In some other embodiments, the gaze tracking application is further configured to store gaze pattern, image data, and metadata from gaze direction information to form a case file.

In another aspect of the present disclosure, there is provided a method of operating a display system. The display system having a display device and a gaze tracking device for tracking gaze direction information of a user. The method of operating the display system comprising displaying an image on the display device; tracking, by a gaze tracking device, gaze direction information indicating where in the image a user is gazing when viewing the image; generating a gaze pattern according to the gaze tracking information; playing back the gaze pattern; and generating focused areas on the display device according to the gaze pattern.

In some other embodiments, the method of operating the display system further comprising tracking at least one additional interaction of the user on a user interface used for displaying and interacting the image.

In some other embodiments, the method of operating the display system further comprising tracking time stamps respectively to the gaze direction information and/or the at least one additional interaction.

In some other embodiments, the method of operating the display system further comprising tracking gaze direction information of another user during playback of the gaze pattern.

In some other embodiments, the method of operating the display system further comprising recording the gaze pattern of the other user for playing back or for comparing with the gaze pattern of the user.

In some other embodiments, the method of operating the display system further 5 comprising displaying a first focused area and only display a second focused area when the gaze tracking application has determined that the other user has focused on the first focused area.

In some other embodiments, the method of operating the display system further comprising display a focused area of the gaze pattern only when the other user has moved focus to another focus area of the gaze pattern without focusing on this focused area of the gaze pattern.

In some other embodiments, the method of operating the display system further comprising filtering the gaze direction information corresponding to unwanted eye movements to reduce interference in the gaze pattern.

In some other embodiments, the method of operating the display system further comprising filtering the image to show the focused area, method of filtering comprising blurring, sharpening, (de)focusing, decoloring, enhancing color saturation, contrast enhancement, and/or contrast reduction.

In some other embodiments, the method of operating the display system further comprising recognizing an Al algorithm as the user; and recording at least one potential area of interest of the Al algorithm as a focused area for the gaze pattern.

In some other embodiments, the method of operating the display system further comprising storing gaze pattern, image data, and metadata from gaze direction information to form a case file.

In some other embodiments, the method of operating the display system further comprising transmitting the gaze pattern to another display system using a transceiver of the display system over a wired or a wireless network; and displaying the gaze pattern on another display device of the other display system.

6 LU103192

In some other embodiments, the method of operating the display system further comprising converting the gaze direction information to at least one gaze coordinate of the gaze pattern or gaze path; sending the at least one gaze coordinate to the other display system for further processing in the other display system to generate a focused area on an image displayed on the other display device.

In some other embodiments, the method of operating the display system further comprising sending an updated image having the focused area with the image to the other display system for direct viewing of the focused area on the other display device.

In some other embodiments, the method of operating the display system further comprising generating an overview window on the other display system, wherein the overview window is configured to display a map of the entirety of the image with an indicator of the relative area of the region of interest on the map.

In some other embodiments, the method of operating the display system further comprising triggering an inputting unit of the display system to activate or deactivate recording of gaze direction information; and/or triggering the inputting unit to activate or deactivate playback of the gaze pattern.

BRIEF DESCRIPTION OF DRAWINGS

These and other features, aspects, and advantages of the apparatus, systems and methods of the present disclosure will become better understood from the following description, appended claims, and accompanying drawing wherein:

Figure 1 illustrates a display system according to some embodiments of the present disclosure;

7 LU103192

Figure 2A and 2B illustrate exemplary images displayed on the display system according to some embodiments of the present disclosure; and

Figure 3 illustrates a flowchart of operation of a gaze tracking application according to some embodiments of the present disclosure.

DESCRIPTION OF EMBODIMENTS

The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the disclosure are shown. This disclosure may, however, be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Like reference numerals refer to like elements throughout.

The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” or “includes” and/or “including” or “has” and/or “having” when used herein, specify the presence of stated features, regions, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, regions, integers, steps, operations, elements, components, and/or groups thereof.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

The present disclosure discloses a gaze tracking application to improve training, efficiency, and quality of diagnosis of medical professionals such as radiologists,

8 LU103192 mammographers, pathologists, dermatologist, etc. The present disclosure may be used for different healthcare applications, such as the following: - Training of medical specialists such as radiologists, mammographers, pathologists, dermatologists, ophthalmologists, etc. - Improving efficiency & performance of medical specialists - Facilitating cooperation between medical specialists

Figure 1 illustrates a display system according to some embodiments of the present disclosure. The display system 1 includes a display device 11, a gaze tracking device 12, a processing device 13, and a memory 14. Optionally, the display system further includes a transceiver 15 configured to transmit and/or receive information from a wired or a wireless network. In some embodiments, the gaze tracking device 12 may be integrated with the display device 11. Alternatively, the gaze tracking device may be a separate device from the display system. Alternative to the display device in FIG. 1, the display device may be direct view display, a projection display, a smart glasses display, a transparent display, a 3D or holographic display, or any other type of equipment that can visualize an image or content.

The processing device 13 includes user interface applications allowing users to interact with the medial images and a gaze tracking application configured to track the gaze direction information indicating where in the image a user is gazing when viewing the image. The gaze tracking application may run in the background when using the user interface application. In some embodiments, the gaze tracking application may generate a gaze pattern using the gaze direction information. The gaze pattern may include focused areas in the image where the user has given more attention to. In some other embodiments, the gaze tracking application may generate a gaze path using the gaze direction information. The gaze path may be a sequence of gaze tracking application that forms a path using sequence of the focused areas where the user has given more attention to. In addition, time stamps for gaze direction information may further be recorded to determine duration for each of the gaze direction information.

The memory 14 is configured to store the gaze pattern, gaze path, and/or additional interaction. In some further embodiments, the images and/or the user interface image may be recorded with the gaze pattern and/or gaze path (step 33 in FIG.3). Optionally,

9 LU103192 the time stamp for each gaze direction information and/or additional interaction may be recorded. In this way, the accuracy of recreating the action of the user with the user interface and the image displayed on the display device in combination with the gaze pattern and/or gaze path during playback may be increased.

In previous embodiments, feedback of gaze patterns or information is typically indicated with arrows, circles or other rather intrusive signs as an overlay on top of the medical images. Such visualization of gaze information/patterns is an intrusive way on top of the medical images and interferes with complex tasks to be performed and reduces performance efficiency. The present disclosure reduces the interference by means of providing using a less intrusive indicators. The present disclosure is suited for applications where complex large images or data structures are involved and where subtle targets and features in the images need to be detected, categorized or interacted with. In the present disclosure, focused areas of the gaze pattern or gaze path are visualized by means of image processing/filtering of the image displayed on the display device.

In some embodiments, during playback of the gaze pattern (step 35 in FIG.3), the focused areas can be indicated by means of highlighting that location. FIGs. 2A and 2B illustrate exemplary images displayed on the display system according to some embodiments of the present disclosure. The luminance of the focused areas 22A and 22B may be increased.

Alternatively, luminance of outside area 21A and 21B of the focused areas may be decrease.

The focused areas may be displayed with a shape having no sharp edges to decrease interference. An exemplary embodiment of showing the focused area is using e.g. spotview of

Barco. Around shape where inside the circular is the focused area 22A and 22B of the image, which is not processed while outside area 21A and 21B of the round shape is dimmed. The border between the round shape and the remaining area of the image can be made as smooth transition between the two areas (i.e. from 22A to 22B) such that no sharp edges is generated.

Creating sharp edges would interfere with the image contents itself as sharp edges would make it more difficult for the human eye to perceive subtle image details. Alternatively, the luminance inside the round shape could be increased, while the luminance outside of the round shape can be dimmed or kept constant. The size of the round shape can be adapted as a function of the image on which the user is performing the task, or based on personal preference. In an exemplary embodiment, if the task to be performed consists of detecting structures, then the size of the round shape may be selected such that the area of the round

10 LU103192 shape is at least it larger than the size of the structures to be detected. Otherwise, the round shape may interfere with the task to be performed. Size of the round shape could also be set (statically or dynamically) based on personal preference. Instead of a round shape includes, but is not limited to, a circular, an elliptical or other shape having little or no sharp edges. On 3D display systems, instead of a flat 2D circle, one could use a 3D sphere. The present disclosure is not limited to a round shape, other patterns or shape having little or no sharp edges may be used for the focused area to reduce interference during display of the focused areas.

In some other embodiments, the gaze direction information as originally received from the gaze tracking device (step 31 in FIG. 3) may be used to generate the gaze pattern and/or gaze path (step 34 in FIG. 3). Optionally, the gaze direction information from the gaze tracking device may be further processed or filtered to generate a refined gaze pattern and/or gaze path (step 32 in FIG. 3). Further processing and filters may be applied to the gaze direction information to enhance or reduce specific eye behavior (such as saccades, smooth pursuit movements, vergence movements, and vestibulo-ocular movements) that may cause unwanted interference in the generated gaze path or gaze pattern. The processing or filtering may include removing or reducing gaze direction information corresponding to saccadic eye movement. In exemplary embodiment, to avoid rapid (sometimes random) change of location of the focused area, filtering gaze direction information corresponding to the rapid or random change may be done such that no or less rapid or random change of focused area will appear on the display when playing back the gaze pattern or gaze path. Further, the filtering could consist of time-averaging recorded gaze direction information of the gaze pattern or the gaze path. Alternatively the filtering or processing comprises of transforming gaze direction information into attention coordinates. In an exemplary embodiment, the attention coordinates may be coordinates of relative areas where the user was paying attention to instead of the individual rapidly changing gaze coordinates. Alternatively, the filtering may include only retaining the gaze direction information corresponding to locations where a minimum time of fixation has taken place. This minimum time of fixation may be set statically or dynamically. Eg. when a radiologist reads a case/image only gaze information for areas where the radiologist at least spent 2% of the reading time can be retained. The minimum time of fixation can be set in such a way that a smooth movement pattern of the circle is obtained.

11 LU103192

Similarly, the other presented filters can be configured in such a way that a smooth movement of the focused area during playback is obtained.

In some other embodiments, during playback of the gaze pattern, filtering operations may be used to highlight the focused area where the user is expected to look at.

Filtering operations include, but are not limited to, blurring, sharpening, (de)focusing, decoloring, enhancing color saturation, contrast enhancement, and/or contrast reduction.

Optionally, blurring may be applied outside the focused area. The degrees of blurring can be adapted (statically or dynamically) for transition between consecutive focused areas to have smooth transition. Optionally, sharpening may be applied on the focused areas. The degrees of sharpening can be adapted (statically or dynamically) for transition between consecutive focused areas to have smooth transition. Optionally, decoloring/desaturation may be applied outside the focused area. The degrees of decoloring can be adapted (statically or dynamically) for transition between consecutive focused areas to have smooth transition.

Decoloring/desaturation is a particularly useful filter for healthcare applications that involve colored medical images. Alternatively, enhancing color saturation may be done on the focused area. Similarly contrast enhancement, or other specific filters that enhance specific image features such as Laplacian pyramid filters, may be applied to the focused areas.

Alternatively, the speed of playing back the gaze patterns or gaze paths may be slower (eg. but not limited to at 95%, 90%, 80%, 70%, 60%, 50%, 40%, 30%, 20%, 10%, etc. of original speed of the expert) or faster (eg. but not limited to at 105%, 110%, 120%, 130%, 150%, 200%, 300%, etc.) than the speed of recording or generating the gaze direction information. In some implementations, the playback timing may be set as desired by the user.

Alternatively, the playback timing may automatically adapted (eg. automatically increasing the playback timing from eg. 10% to 100%as a junior user gains more experience or achieves a higher performance level). When automatically adapting the timing behavior care needs to be taken that these adaptations are done in a smooth way to avoid abrupt changes that can interfere with the task being performed.

Medical images are often not looked at in a static way. An example, CT images consist of a stack of individual slices and a radiologist will browse through this stack by means of keyboard, mouse interactions or by means of other types of input devices or man-machine interfaces. Another example: pathology images are very high resolution images and the human

12 LU103192 eye can never observe all small details in those images when visualizing an entire slide image at once. Instead, pathologists are extensively making use of panning and zooming operations to inspect parts of the image at higher zoom levels and in more detail. For 3D content users are often rotating the 3D content or bringing the content closer or further away.

In general more and more “processing filters”, “image enhancement filters”, “image analysis functionality”, “Al functionality” is being used by medical specialists. Examples include, but are not limited to, window/level operations, contrast enhancement filters, color processing filters, segmentation operations, measurement or analysis operations/functionality, etc. All such interactions also are typically done by means of keyboard, mouse interactions or by means of other types of input devices or man-machine interfaces.

While the user is performing a task, additional interaction with the image may be performed. The additional interaction further includes, but are not limited to, eg. panning & zooming actions, filtering actions (eg. contrast enhancement, brightness changes, change of window-level settings), segmentation or processing actions, invoking of tools such as Al tools or image analysis tools, activating commands, voice or gesture commands, etc. In further embodiments, the gaze tracking application further records the additional interaction in the memory 14. Recording the additional interaction may be done by recording keyboard or mouse strokes or commands to/from the used human-machine interface devices (mouse, keyboard, dedicated user interaction devices, speech recognition, gesture recognition, ...).

Playing back the additional interactions may be done by generating the same recorded commands again or replaying the instructions that were triggered by the original interactions.

Optionally, the additional interactions may be recorded with corresponding time stamps. The time stamps may be used to indicate the location of the gaze pattern or gaze path where the additional interactions are implemented. For example: the gaze tracking application is used to record the user performing a task, at timing 21 seconds (after starting the task) the expert performed a pan/zoom action, and at timing 32 seconds (after starting the task) the user performed a window/level operation. When the other user playing back the gaze pattern or the gaze path, the focused areas may be displayed and, in addition, the additional interactions that the user has performed may be automatically triggered as well. In the example, given that 21 seconds after the starting the playback, the same pan/zoom action will

13 LU103192 be automatically triggered. And, 32 seconds after, the same window/level action will be automatically triggered.

In case the playback is accelerated or slowed-down, then timing behavior of the additional interaction needs to remain consistent. During playback, the gaze pattern or gaze path need to remain synchronized with the execution of the additional interactions. When playback is done in half speed, the additional interaction needs to be done in half speed as well. Following the above example, the c pan/zoom action would be executed at 10.5 seconds (instead of 21 seconds), and window/level action will be triggered at 16 seconds (instead of at 32 seconds).

In an exemplary embodiment, when generating the gaze path or gaze pattern, the attention of the user remains at a location 1 for 4 seconds, then a window-level action is performed, then the focused area remains another 3 seconds on location 1, then focused area moves on to location 2. In this situation, during payback, the automatic activation of the window-level action will be done after focused area of the gaze path or gaze pattern remains on location 1 for 4 seconds. After executing the window-level action, the focused area will remain for another 3 seconds on location 1, and then the focused area will automatically move to location 2.

Further, gaze tacking application may further be configured to perform gaze tracking on another user during playback of the gaze pattern or the gaze path. Instead of directly displaying the focused areas of the stored gaze path or gaze pattern, the gaze path or gaze pattern are used as a guide or correction tool for the other user. In some embodiments, the focused areas of the gaze pattern or gaze path are used as target points to direct the attention of the other user. When a first focused area is shown, the gaze direction of the other user may be measured. Then, the gaze direction information of the other user is matched with the first focused area. When the gaze direction information of the other user are matched with the first focused area, a second focused area may be displayed for the other to target next.

Alternatively, the other user may proceed to perform a task. Only when the gaze pattern or gaze path of the other user deviates from the stored gaze path or gaze pattern, then the focused area of the stored gaze path or gaze pattern may be shown to redirect the gaze direction of the other user. Alternatively, the gaze path or gaze pattern of the other user is

14 LU103192 only compare with the stored gaze path or gaze patten when the other user has completed the task to determine whether the users have different focused areas.

In an exemplary embodiment, when the other user is using the gaze tracking application, the automatic activation of the window-level action will be done once gaze tracking device has detected that the other user has changed gaze direction after focusing on location 1 (for a duration shorter than 4 seconds or alternatively longer than 4 seconds). After executing the window-level action, the focused area presented to the other user will remain on location 1. When the other user changes gaze direction after focusing on location 1 (for a duration shorter than 3 seconds or alternatively longer than 3 seconds), the focus area automatically move to location 2.

The gaze tracking application further includes an inputting unit to start a recording or start a play back session when triggered. The inputting unit may be implemented using buttons, entering a command, clicking on an icon or performing another Ul user interaction, voice recognition, gesture recognition, and so on. Alternatively, additional way is to create a label/sign on the bezel of the display area (“virtual buttons”) and using gaze tracking device to trigger the actions by looking at these “virtual buttons”. The gaze tracking device can detect when a user gazes on a virtual button and then can activate the appropriate corresponding command. These labels/signs can be static signs (eg. stickers) or dynamically changeable signs created eg. by a mini display. Such dynamic signs/labels allows for easy (re)configuration of actions that a user commonly triggers.

Training of medical specialists

Diagnosing medical images is a very complex task. For residents and junior radiologists/pathologists, it is not obvious at all to know how one can read large complex medical images in a quick efficient way, while still achieving high quality of diagnosis and not missing any clinically relevant findings.

Today this is mostly being learned by means of interactive training. A senior radiologist is verbally explaining how images “should be read”. The senior radiologists describes the different areas of the image that he/she sequentially looks at and what is visible in those image areas. The resident then tries to mimic that reading behavior by trying to follow

15 LU103192 the same reading pattern. This is clearly not an ideal training solution, as it is very difficult to describe a reading strategy. Sometimes the expert has problems to even know how trainees is reading because it is based on experience and has become a habit.

The present disclosure uses gaze tracking to measure/record the reading patterns of expert radiologists on a variety of medical images to provide a more reliable data that describes how experts read medical images. The recorded gaze pattern or gaze paths then can be shown to trainees/residents to better explain how reading should be done. The trainees can use the same gaze tracking setup and have gaze patterns or gaze paths be recorded, such that after the reading session the gaze patterns or gaze paths of the expert and the trainee can be compared.

When using the display system of FIG. 1 as described, a user interface application used for interaction with complex medical image allowing the user to interact with the image data is activated on the display system. Optionally, the gaze direction application may be activated by triggering an inputting unit to start recording the interaction of the user with the display system. While the user such as an expert is performing a task by interacting with image data, the gaze direction information measured corresponding to the gaze direction of the user may be recorded in real time. The gaze direction information may correspond to areas of the displayed medical image where the user given attention to. The areas may be combined to form a gaze pattern or a gaze path. The gaze pattern may indicate individual areas where the user has given attention to. The gaze path may further indicate a sequence of the individual areas where the user has given attention to. Further, the additional interaction made by the user to the image data is further recorded and stored in the memory. Optionally, the timestamps for each gaze direction information and/or additional interaction may be correspondingly recorded. In this way, the accuracy of recreating the interaction of the expert with the image data for training a trainee may be increased.

After, another user such as a trainee may use the stored gaze path or gaze pattern to trace the interaction of the expert user with the image data. Additionally, the other user may further trace additional interactions to further determine changes in displayed image data when the expert was performing the recorded task. Optionally, the playback of the gaze path, gaze pattern, and/or additional interactions is activated by triggering an inputting unit of the gaze tracking application.

16 LU103192

Alternatively, when the gaze path, gaze pattern, and/or additional interactions makes up a generic process to be performed on a medical image, the playback of the gaze path, gaze pattern, and/or additional interactions is not limited to being repeated on the same medical image. The gaze path, gaze pattern, and/or additional interactions may be played back to interact with a different medical image. For example: a radiologist may always read a breast cancer screening mammography image in the same way. This means that the radiologist will in a consistent way inspect such image by sequentially looking at relevant parts (clinical structures in the image) of the image. In such a situation, the playback of the gaze path can be done on a different image by directing the focused areas to those relevant image parts in the correct order.

During playback, focused areas are generated according to thegaze path or gaze pattern. The focused areas are played backed to have the same timing as the gaze direction information have been recorded. After a first time period corresponding to a first focused area has passed, a second focused area is displayed on the display area for a second time period.

Optionally, the additional interactions may be automatically performed on the user interface by the gaze tracking application or the gaze tracking information may present an indicator for the other user to follow. The additional interactions are inserted in the playback at the same time stamps as during recording of the gaze direction information. In this way, the trainee will be able to see the step by step process as how the expert has perform the task on the same image data. As the trainee gains more experience, the playback speed can be increased. Eg. the first few training sessions of the trainee could be done at 10% of the speed of the expert, and playback speed can then be gradually increased to eg. 80% of the expert towards the end of the trainee’s training schedule.

Alternatively, the gaze tracking application may serve as a step by step guide for the trainee. Each of the focused area of the gaze pattern may be displayed on the display device. After the trainee has focused on the first focused area, a second focused area may be displayed on the display device. The process will continue until all focused areas of the gaze pattern or gaze path has been completely viewed by the trainee. In the same way, an indication corresponding to additional interactions may also be indicated for trainee to perform during playback. In this way, the trainee may be able to get hands on training while performing a task and not merely a video playback of the task performed.

17 LU103192

Alternatively, only when the trainee has focused on a same area as the focused area of the gaze pattern or gaze path will the focused area be displayed on the display device.

In this way, a trainee will be able to verify or test the accuracy of the knowledge he has to perform the task. The trainer may also use the gaze tracking application as a test to see progress of the trainee without influencing the decision of the trainee while performing the task.

Optionally, the gaze direction information of the trainee may be further recorded by the gaze tracking application and be allowed to show the trainee how he performed the task. In this way, the trainee will be able to determines his accuracy and/or efficiency to perform the task. Further, the trainee may be able to use different settings for viewing playback, such as settings for focused area and/or timing setting as mentioned above. The abovementioned features of the gaze tracking application may be adjusted or edited accordingly to fit the need of the trainee or to fit the level of knowledge of the trainee.

Facilitating cooperation between medical specialists

Medical specialists often consult each other for difficult cases. In such situation, specialists being in different locations may need to discuss a case. A common practice is to verbally provide instructions on where to look at in the case file and verbally describe what is seen by one another.

In some embodiments, when sharing gaze pattern or gaze path between two or more users, a first user being the user sharing information may be using the display system of the present disclosure while a second user may be using a normal display system without a gaze tracking device or another display system of the present disclosure. During consultation the users may be looking at the same image data on respective display devices. The gaze direction information of the first user is tracked using gaze tracking device. The gaze tracking application is configured to convert the gaze direction information to at least one gaze coordinate of the gaze pattern or gaze path. Through the tranceiver of the display system, the at least one gaze coordinate is sent to the display system of the second user for further processing to generate a focused area on the image displayed. Alternatively, an updated image having a focused area is sent to the display system of the second user for direct viewing of the second user. Optionally, the image displayed on the display device of the first user does not

18 LU103192 have the focused area displayed. In this way, the first user does not notice any visual changes on the displayed image.

Optionally, the first user is able to control image data displayed on the display system of the second user using the user interface. The command of the additional interaction on the display system of the first user is sent to the display system of the second user to control the image data of on the display system of the second user. In this way, the first user and the second user is guaranteed to be viewing the same image data in real time.

Optionally, the gaze tracking application is further installed in the display system of the second user. While in a sharing state, the second user may use the gaze tracking application to deactivate the display of the focused area. In this way, the second user will be able to have a clear view of the areas outside the focused area shared by the first user.

The gaze tracking application further allows the second user to also share at least one gaze coordinate or focused area to the first user. In this way, the sharing may be interactive instead of one way.

In some further embodiments, when the display systems of the first user and the second user have different display resolutions, the image data may either be scaled to match for the at least two display systems, show only a portion of the image data on the display system having lower resolution, or a combination of both. For example, if display system of the first user has a display device having a 5 Megapixel resolution and the display system of the second user only has a display device only having 3 Megapixel resolution, the image data from the display device of the first user may not be fully displayed on the display system of the second user. One option is to first downscale the image data from 5 MegaPixel resolution to 3

MegaPixel resolution. However, quality problems may arise as subtle image details may be lost because of the scaling. Another option is to select a 3 MegaPixel region of interest inside the 5

MegaPixel image, and only transmit/visualize the region of interest to the 3 MegaPixel display.

However, the selection of the region of interest may result in an additional action from the user sharing the image data. In some embodiments, the gaze tracking application may be used to determine region of interest according to the gaze direction information. The gaze tracking application region of interest shown in the display device having lower resolution may be determine according to the gaze coordinate of determined corresponding to the gaze direction information of the first user. Optionally, the gaze tracking application may further provide an

19 LU103192 overview window for the display device having lower resolution. The overview window is configured to display a map of the entirety of the image data with an indicator of the relative area of the region on interest on the map. In this way, the user of the display device with lower resolution would still be able to know which part of the image is being shown in the region of interest.

Improving efficiency & performance of medical specialists

Al algorithms are increasingly used in healthcare applications. One of the problems with current Al algorithms is the high number of false positives. A false positive means that the patient is not sick (eg. no breast cancer) but the Al algorithm predicts that the patient is sick (eg. breast cancer present).

Taking cancer screening as an example, false positives are especially problematic in a screening context (where probability of a disease is low) as the Al algorithm would mark a large proportion of cases as malignant, while in fact there is only a very small number of malignant cases. Incorrectly marking a case as malignant often triggers expensive, painful and complex procedures to examine the patient. This also creates a lot of anxiety with the patient.

Therefore, too many false positives is not acceptable.

When radiologists are using Al algorithms that have many false positives then this results into a non-working solution. In theory a radiologist could try to ignore the many false positives of the Al algorithm but in practice this is not feasible. First of all, the mere fact that an

Al algorithm puts a mark on an image and states that this “may be a malignancy” is creating already a bias and doubt with the radiologist and makes it more likely that the radiologist also believes it is a malignant case. Secondly, from a medico legal point of view it is extremely difficult for a radiologist to state that an image is benign if an Al algorithm has marked a potential malignancy on the image. Because this may trigger legal claims against the radiologist in case in the end the patient had a malignancy after all.

The present disclosure may have an Al algorithm as a first user where gaze pattern or gaze paths may be recorded by the gaze tracking applciation. In this way, the potential area of interest found by the Al algorithm may be recorded in the memory. A second user may later on (after the Al algorithm has gone through the image data) or at the same time (as the Al algorithm is running in the background) go through the image data. While the

20 LU103192 second user is going through the image data, the gaze tracking application is used to track the gaze direction information of the second user. When the gaze tracking application determines that the second user has not or not sufficiently (eg. not long enough in relative or absolute terms) looked at the potential area of interest, a focused area of the potential area of interest shall be displayed for the second user to see. In this way, while Al algorithm is still acting as a secondary opinion, it is not influencing the behavior of the primary user, such as a radiologist, unless there is high likelihood that the radiologist has not even looked at / considered the particular relevant part of the image for which is a potential area of interest for the Al algorithm.

Additionally, the time stamps of the gaze tracking application may be used to determine if enough attention has been given to areas of the image. Further, the gaze tracking application may further use gaze direction information to measure other metrics, eg. the level of fatigue of a user by determining how quickly the user is going through the image or how much unwanted gaze direction information is being filtered by the gaze tracking application. In case there is high(er) level of fatigue, the gaze tracking application may display the potential area of interest of the Al algorithm may be shown more quickly, or a warning may be given to the user about fatigue state.

Optionally, the gaze path, gaze pattern, and/or additional interactions are extracted stored with the image data to form a case file. Additional information corresponding to the analysis done is further determined and stored as metadata or headers of the stored case file. Additional information may include information on how long the user inspected the image data, level of fatigue, what particular areas in the image were looked at etc. The case file could serve as evidence in case there would be a dispute on whether the radiologist has properly performed the task or potentially made a medical error resulting into harm for the patient. The logging of the gaze information and the way the case was read is particularly relevant for large complex cases such as eg. complex pathology cases consisting of many individual slide images that each are high resolution images requiring a lot of pan/zoom operations and where it is almost impossible to inspect all parts of the image. Another example where this is particularly useful is for large 3D datasets (such as CT, MRI, DBT images) where the specialist needs to browse through a large stack of images and it is not always clear on whether everything has been looked at.

21 LU103192

22 LU103192

In some other embodiments, the gaze tracking application further recognizing an

Al algorithm as a focused area for the gaze pattern.

23 LU103192

In some other embodiments, the method of operating the display system further comprising displaying a first focused area and only display a second focused area when the gaze tracking application has determined that the other user has focused on the first focused area.

24 LU103192

25 LU103192

While the invention has been described hereinabove with reference to specific embodiments, this was done to clarify and not to limit the invention. The skilled person will appreciate that various modifications and different combinations of disclosed features are possible without departing from the scope of the invention.

Claims

26 LU103192 CLAIMS

1. A display system, comprising: a display device configured to display an image; a gaze tracking device configured to track gaze direction information indicating where in the image a user is gazing when viewing the image; a processing device configured to generate a gaze pattern using the gaze direction information; and a memory configured to store the gaze pattern; wherein when playing back the gaze pattern, one or more focused areas on the image are generated according to the gaze pattern to identify areas viewed by the user.

2. The system of claim 1, wherein the gaze pattern is played back in another display device of another display system.

3. The system of claim 2, further comprising a transceiver configured to transmit the gaze pattern to the other display device using a wired or a wireless network.

4. The system of any of claims 1 to 3, wherein the gaze tracking application is configured to convert the gaze direction information to at least one gaze coordinate of the gaze pattern or gaze path, wherein the at least one gaze coordinate is sent to the other display system through the transceiver to generate a focused area on an image displayed on the other display device.

5. The system of any of claims 1 to 3, wherein an updated image having the focused area with the image is sent to the other display system for direct viewing of the focused area.

6. The system of any of the preceding claims, wherein when display resolutions of the two display systems are different, the image transmitted is either scaled to match for the two display systems, show only a portion of the image on the display system having lower resolution, or a combination of both, wherein the display system having lower resolution, at least displays a region of interest where the user is focused on.

7. The system of claim 6, wherein the gaze tracking application is configured to generate an overview window, the overview window is configured to display a map of the entirety of the image with an indicator of the relative area of the region of interest on the map.

27 LU103192

8. The system of any of the preceding claims, wherein a normal speed of playback of the gaze pattern is according to a normal speed of the user going through the image, wherein the speed of the playback is controllable to speed up or slow down.

9. The system of any of the preceding claims, wherein the processing device further comprising a user interface application for controlling the image displayed on the display device, the memory further configured to store additional interactions comprising input to the user interface application when generating the gaze pattern.

10. The system of any of the preceding claims, wherein the gaze tracking application further configured to filter the gaze direction information corresponding to unwanted eye movements to reduce interference in the gaze pattern.

11. The system of any of the preceding claims, wherein the gaze tracking application furthered configured to filter the image to show the focused area, method of filtering comprising blurring, sharpening, (de)focusing, decoloring, enhancing color saturation, contrast enhancement, and/or contrast reduction.

12. The system of any of the preceding claims, wherein the gaze tracking application includes an inputting unit configured to activate or deactivate recording of gaze direction information and/or activate or deactivate playback of the gaze pattern.

13. The system of any of the preceding claims, wherein the gaze tracking application further recognizing an Al algorithm as the user and configured to record at least one potential area of interest of the Al algorithm as a focused area for the gaze pattern.

14. The system of any of the preceding claims, wherein the gaze tracking application is further configured to track gaze direction information of another user while playing back the gaze pattern.

15. The system of claim 14, wherein the gaze tracking application is further configured to record the gaze pattern of the other user for playing back or for comparing with the gaze pattern of the user.

16. The system of claim 14, wherein the gaze tracking application is further configured to display a first focused area and only display a second focused area when the gaze tracking application has determined that the other user has focused on the first focused area.

17. The system of claim 16, wherein the second focused area is displayed after focusing on the first focused area for an expected time period.

28 LU103192

18. The system of claim 14, wherein the gaze tracking application is further configured to display a focused area of the gaze pattern only when the other user has moved focus to a latter focus area of the gaze pattern without focusing on this focused area of the gaze pattern.

19. The system of any of the preceding claims, wherein the gaze tracking application is configured to record the gaze pattern with corresponding time stamps.

20. The system of any of the preceding claims, wherein the gaze tracking application is further configured to store gaze pattern, image data, and metadata from gaze direction information to form a case file.

21. A method of operating a display system having a display device and a gaze tracking device for tracking gaze direction information of a user, comprising: displaying an image on the display device; tracking, by a gaze tracking device, gaze direction information indicating where in the image a user is gazing when viewing the image; generating a gaze pattern according to the gaze tracking information; playing back the gaze pattern; and generating focused areas on the display device according to the gaze pattern.

22. The method of claim 21, further comprising: tracking at least one additional interaction of the user on a user interface used for displaying and interacting the image.

23. The method of any of claims 21 and 22, further comprising: tracking time stamps respectively to the gaze direction information and/or the at least one additional interaction.

24. The method of any of claims 21 to 23, further comprising: tracking gaze direction information of another user during playback of the gaze pattern.

25. The method of claim 24, further comprising: recording the gaze pattern of the other user for playing back or for comparing with the gaze pattern of the user.

26. The method of claim 24, further comprising: displaying a first focused area and only display a second focused area when the gaze tracking application has determined that the other user has focused on the first focused area.

29 LU103192

27. The method of claim 26, wherein the second focused area is displayed after focusing on the first focused area for an expected time period

28. The method of claim 24, further comprising: display a focused area of the gaze pattern only when the other user has moved focus to another focus area of the gaze pattern without focusing on this focused area of the gaze pattern.

29. The method of any of claims 21 to 28, further comprising: filtering the gaze direction information corresponding to unwanted eye movements to reduce interference in the gaze pattern.

30. The method of any of claims 21 to 29, further comprising: filtering the image to show the focused area, method of filtering comprising blurring, sharpening, (de)focusing, decoloring, enhancing color saturation, contrast enhancement, and/or contrast reduction.

31. The method of any of claims 21 to 30, further comprising: recognizing an Al algorithm as the user; and recording at least one potential area of interest of the Al algorithm as a focused area for the gaze pattern.

32. The method of any of claims 21 to 31, further comprising: storing gaze pattern, image data, and metadata from gaze direction information to form a case file.

33. The method of any of claims 21 to 32, further comprising: transmitting the gaze pattern to another display system using a transceiver of the display system over a wired or a wireless network; and displaying the gaze pattern on another display device of the other display system.

34. The method of any of claims 21 to 33, further comprising: converting the gaze direction information to at least one gaze coordinate of the gaze pattern or gaze path; sending the at least one gaze coordinate to the other display system for further processing in the other display system to generate a focused area on an image displayed on the other display device.

35. The method of any of claims 21 to 33, further comprising: sending an updated image having the focused area with the image to the other display system for direct viewing of the focused area on the other display device.

30 LU103192

36. The method of any of claims 21 to 35, wherein when display resolutions of the two display systems are different, the image transmitted is either scaled to match for the two display systems, show only a portion of the image on the display system having lower resolution, or a combination of both, wherein the display system having lower resolution, at least displays a region of interest where the user is focused on.

37. The method of any of claims 21 to 36, further comprising: generating an overview window on the other display system, wherein the overview window is configured to display a map of the entirety of the image with an indicator of the relative area of the region of interest on the map.

38. The method of any of claims 21 to 37, wherein a normal speed of playback of the gaze pattern is according to a normal speed of the user going through the image, wherein the speed of the playback is controllable to speed up or slow down.

39. The method of any of claims 21 to 38, further comprising: triggering an inputting unit of the display system to activate or deactivate recording of gaze direction information; and/or triggering the inputting unit to activate or deactivate playback of the gaze pattern.