JP7616372B2 - Vehicle display system, vehicle display method, and vehicle display program - Google Patents

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on Japanese Application No. 2021-092992, filed on June 2, 2021, the contents of which are incorporated herein by reference.

The present disclosure relates to a vehicle display system, a vehicle display method, and a vehicle display program.

Many peripheral monitoring functions using peripheral cameras have been proposed to supplement the visual information required for driving. For example, see Patent Document 1. In the technology described in Patent Document 1, images captured by peripheral cameras at the rear and sides are processed into a continuous composite image, making multiple images appear as a single image. Electronic mirror technology has also been provided that makes it easier to monitor the surroundings by understanding the direction of turning using blinkers and widening the angle of view in the direction of turning.

In this way, by processing multiple images into one continuous image or by mechanically changing the angle of view on the camera side, the occupants can see the image they are supposed to see without changing the location they are looking at. These issues can also be solved by framing the display while increasing the display area of the display, but framing technology alone cannot provide the information that vehicle occupants want.

There is also technology being offered that supports driving by converting information obtained from various sensors into graphical visual information that is easy to see and understand. As technology improves, the amount of information handled within vehicles increases, and the size of display screens and the number of displays installed are on the rise.

Although a large amount of information can be displayed to support vehicle occupants, the occupants' field of vision is limited and they are unable to view all the information at once. This increases the number of times that vehicle occupants move their eyes or turn their heads, which is inconvenient. In addition, when a large amount of information is displayed, it is difficult to quickly search for and view the necessary and appropriate information at a glance.

JP 2019-118016 A

The object of the present disclosure is to provide a vehicle display system, a vehicle display method, and a vehicle display program that enable occupants to check appropriate information while minimizing the number of times they move their eyes or shake their heads.

According to one aspect of the present disclosure, the determination unit determines whether the primary content being viewed by the vehicle occupant is the map screen , the change detection unit determines that the vehicle's traveling speed has rapidly decreased within a predetermined time from the vehicle's speed signal detected by the vehicle speed sensor, and determines that the number of other vehicles has increased around the vehicle by detecting a change in the situation of the image captured by the backup camera as related content related to the map screen. The display processing unit changes the display mode of the image captured by the backup camera based on the detection result by the change detection unit, and displays the map screen and the image captured by the backup camera in the field of view ahead of the vehicle occupant's line of sight .

According to one aspect of the present disclosure, the map screen and the image captured by the backup camera are displayed in the field of view of the vehicle occupant's line of sight based on the detection result by the change detection unit, so that the vehicle occupant can check the image captured by the backup camera along with the map screen . This allows the occupant to check appropriate information while minimizing the number of times they move their line of sight or shake their head.

The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:
FIG. 1 is a diagram showing a control image of the first embodiment; FIG. 2 is a block diagram showing a vehicle display system; FIG. 3 is a flowchart showing the outline of the process. FIG. 4 is an illustration of a display as seen by a vehicle occupant; FIG. 5 shows a first example of a display mode. FIG. 6 is a bird's-eye view showing the positional relationship between the host vehicle and an obstacle; FIG. 7 shows a second example of the display mode. FIG. 8 is an illustration of a display as seen by a vehicle occupant; FIG. 9 shows a third example of the display mode. FIG. 10 shows a fourth example of the display mode. FIG. 11 is a first explanatory diagram of the change in the display mode. FIG. 12 is a second explanatory diagram of the change in the display mode. FIG. 13 is a fifth example of a display mode in the second embodiment. FIG. 14 is a sixth example of the display mode. FIG. 15 is a seventh example of a display mode. FIG. 16 is an eighth example of the display mode.

Below, several embodiments of the vehicle display system 1, the vehicle display method, and the vehicle display program will be described with reference to the drawings. Note that parts that are essentially common to each embodiment will be described with the same reference numerals.

As shown in Figure 1, the vehicle display system 1 is composed of a cockpit system equipped with multiple displays, such as a center display 3 and an electronic mirror 4. However, the number, installation, and configuration of the displays are merely examples and are not limited to these.

The center display 3 is composed of, for example, an LCD display or an organic EL display, and is installed between the driver's seat and the passenger seat.

The center display 3 is located near the center console where it is easily visible to both the driver and the passenger in the front passenger seat, and is capable of displaying various contents. An operation panel 21 is configured above the center display 3, and allows the selection of contents to be displayed on the center display 3, as well as input operations for operating the air conditioning, audio, and navigation functions.

The electronic mirror 4 is composed of a door mirror camera 4a and a display 4b. See Figure 2. The door mirror camera 4a is a peripheral camera 23 that captures an image of the rear of the vehicle Ca, i.e., the side rear, from the location where the door mirror of the vehicle Ca is installed.

The display 4b of the electronic mirror 4 is, for example, a liquid crystal display or an organic electroluminescence display. The displays 4b are installed one on each side of the driver, for example, at the bottom of the A-pillar inside the vehicle.

The display 4b of the electronic mirror 4 is positioned to allow the driver to easily check the area behind the vehicle, and the display 4b is configured to display information about the vehicle's surroundings captured by the door mirror camera 4a.

As shown in Figures 1 and 2, many ECUs 5 are configured within the vehicle and connected to an in-vehicle network 25. The ECUs 5 include a display system ECU, a surroundings monitoring system ECU, a driving control system ECU, and a DCM that communicates with the outside of the vehicle. DCM stands for Data Communication Module. The driving control system ECUs include the well-known vehicle control ECU, engine control ECU, motor control ECU, brake control ECU, steering control ECU, and integrated control ECU. The driving control system ECUs include an autonomous driving ECU. The autonomous driving ECU is an Autonomous Driving Electric Control Unit.

When the autonomous driving ECU receives an automatic control signal, it drives the driving actuator to perform a corresponding predetermined level of driving assistance or autonomous driving. For example, level I driving assistance can perform automatic braking to avoid collisions with obstacles, following driving to follow the vehicle in front, or lane prevention driving to prevent the vehicle from going out of the lanes on either side. Level II autonomous driving can perform a combination of level I driving assistance, or autonomous driving under specific conditions, such as an autonomous driving mode that automatically overtakes slower vehicles on a highway or automatically merges onto the highway. In autonomous driving at level III or higher, the system performs all driving tasks while monitoring the vehicle.

Each ECU 5 is mainly composed of a microcomputer equipped with various storage units 6 such as a processor, cache memory, RAM, ROM, I/O, and a bus connecting these. Each ECU 5 is communicatively connected to other ECUs 5 provided in the vehicle via a communication control unit 7 and an in-vehicle network 25.

In this embodiment, as shown in FIG. 2, multiple display system ECUs 5 constitute an HCU as a vehicle device 10, and the processing power of the internal physical resources of the display system ECUs 5 is shared to perform display processing on the center display 3 and the display 4b of the electronic mirror 4. The display system ECUs 5 may be connected to each other by dedicated lines, or may be constituted by a single ECU 5. HCU stands for Human Machine Interface Control Unit and indicates an information presentation control device. The aforementioned memory unit 6 indicates a non-transitive substantial storage medium that non-temporarily stores programs and data that can be read by a computer. The non-transitive substantial storage medium is realized by a semiconductor memory or the like.

As shown in FIG. 2, the vehicle device 10 includes a control device 11, a calculation device 12, a memory unit 6, a display processing unit 13, an audio processing unit 14, an I/O control unit 15 that manages input and output from various devices, and a communication control unit 7 that manages communications with other ECUs 5.

In addition, the output signals of the position detector 19, operation panel 21, occupant monitor 22, surrounding camera 23, and distance detection sensor 24 shown in Figure 2 will be described as being input through the I/O control unit 15, but signals may also be input from other ECUs 5, such as an ECU for surrounding monitoring or an ECU for driving control, via the in-vehicle network 25.

Based on the control of the control device 11, the calculation device 12 calculates the display area to be displayed on the display screen of the center display 3 and the display 4b of the electronic mirror 4 for content such as images, sentences, characters, etc. (hereinafter referred to as images, etc.) stored in the memory unit 6, calculates in which area of the display screen of the center display 3 and the display 4b of the electronic mirror 4 the content such as images will be displayed, and on which area the content such as images will be superimposed, and outputs the display area together with the content such as images to the display processing unit 13 via the control device 11.

The display processing unit 13, under the control of the control unit 11, processes and displays content such as images in the aforementioned display areas on the display screens of the center display 3 and the display 4b of the electronic mirror 4. This allows content such as images to be displayed superimposed on each display layer on the display screen of each display 3, 4b. The audio processing unit 14, under the control of the control unit 11, inputs received voice input from the microphone 17 and outputs transmitted voice from the speaker 18. When the audio processing unit 14 inputs text or character content from the control unit 11, it converts it into voice and reads it out through the speaker 18.

The position detector 19 detects the position with high accuracy using a GNSS receiver such as a well-known GPS (not shown) and inertial sensors such as an acceleration sensor and a gyro sensor. The position detector 19 outputs a position detection signal to the control device 11 through the I/O control unit 15. The position identification unit 11a of the control device 11 realizes the function of an ADAS locator that sequentially measures the current position of the vehicle with high accuracy based on map information input from a map data input device and the position detection signal of the position detector 19. ADAS is an abbreviation for Advanced Driver Assistance Systems.

In this case, the vehicle position is represented in a coordinate system consisting of latitude and longitude, where, for example, the X-axis indicates longitude and the Y-axis indicates latitude. The vehicle position may be measured using, for example, a vehicle speed sensor mounted on the host vehicle Ca. The vehicle speed sensor detects the vehicle speed and outputs a speed signal to the control device 11. Positioning may be performed based on information on the travel distance obtained based on the sensing results of the vehicle speed sensor, and various configurations may be adopted as long as the position of the host vehicle Ca can be identified. The control device 11 can perform so-called navigation processing based on the current position of the host vehicle Ca.

The operation panel 21 is a touch panel configured on the center display 3, and when an operation input is made by the occupant, the I/O control unit 15 accepts the operation input and outputs it to the control device 11. The control device 11 executes control based on the operation signal of the operation panel 21.

The occupant monitor 22 detects the state or operation state of the occupant in the vehicle. The occupant monitor 22 is configured using, for example, a power switch, an occupant state monitor, a turn switch, an automatic control switch, etc., and outputs various signals to the control device 11. The occupant monitor 22 may include a steering sensor that detects whether the steering wheel is being gripped or steered by the driver, a seating sensor that detects whether the driver is seated in the seat, an accelerator pedal or brake pedal depression sensor, etc.

When the power switch is turned on by an occupant in the vehicle cabin to start the internal combustion engine or electric motor, it outputs a signal corresponding to the operation. The occupant status monitor includes a camera that detects the status of an occupant in the D or P seat by photographing the status of the occupant with an image sensor and outputs an image signal. The driver's occupant status monitor is called a DSM. DSM is an abbreviation for Driver Status Monitor. The occupant status monitor shines near-infrared light on the driver's head to obtain an image signal, performs image analysis as necessary, and outputs the image to the control device 11.

The occupant status monitor is used to detect the status of occupants such as the driver, particularly during driving assistance or autonomous driving. The turn switch is turned on by an occupant in the vehicle cabin when activating the turn signal of the host vehicle Ca, and outputs a turn signal to turn right or left in response to the operation.

When an occupant in the vehicle turns on the automatic control switch in order to command a specific driving assistance or automatic control for the vehicle's driving state, the automatic control switch outputs an automatic control signal corresponding to the operation. This automatic control signal is output to the ECU of the driving control system, thereby enabling the corresponding specific level of driving assistance or automatic driving control to be executed.

The control device 11 can determine the behavior of the vehicle occupant, for example the direction in which they are looking, based on the sensor signal from the occupant monitor 22, and can also input the operation status of the power switch, the operation status of the turn signal, command information for the automatic control of the vehicle, and speed information from the vehicle's speed sensor.

The peripheral cameras 23 include a forward visibility camera that images the area in front of the vehicle, a rear camera that images the area behind the vehicle, a corner camera that images the front and rear parts of the vehicle, a side camera that images the side of the vehicle, and a door mirror camera 4a of the electronic mirror 4, and these are output to the control device 11 via the I/O control unit 15 as respective image signals of the front guide monitor, rear guide monitor, corner view monitor, side guide monitor, and electronic mirror 4, and stored in the memory unit 6.

The display 4b of the electronic mirror 4 displays the image captured by the door mirror camera 4a. The electronic mirror 4 reproduces a rearview mirror. Therefore, the electronic mirror 4 displays a mirror-inverted image of the image captured by the door mirror camera 4a on the display 4b for the electronic mirror 4.

The vehicle is also equipped with a distance detection sensor 24 as a perimeter monitoring sensor that detects the distance to an obstacle. The distance detection sensor 24 is composed of a clearance sonar 24a, LiDAR, or radar using millimeter waves, and detects the distance to an obstacle such as a vehicle or person approaching the front, front side, rear side, rear, or side of the vehicle.

Each ECU 5 is equipped with a SoC. SoC stands for System On Chip. The above-mentioned microcomputer is built into the mounted SoC. The microcomputer built into the SoC of the ECU 5 is configured to run a variety of applications (hereafter referred to as apps) on a pre-installed general-purpose OS and real-time OS, thereby enabling the control device 11 to realize the functions described below. The apps include an image processing app and other apps. In response to a drawing request from the image processing app, the processor built into the SoC performs drawing processing on the display screen of the center display 3 and the display 4b of the electronic mirror 4.

Furthermore, the content rendered by the app can also be displayed on other displays, in this case the center display 3. The apps include navigation apps, etc. The navigation app realizes the navigation function and renders a navigation screen including a map image and the current position of the vehicle, which are mainly displayed on the center display 3 or the display 4b of the electronic mirror 4.

The apps also include an image synthesis app. The image synthesis app is an app that specifies the size and type of various image content to be displayed on the display, synthesizes the images of the image content within one frame, and outputs this synthesized mixed image to the center display 3 or the display 4b of the electronic mirror 4. The image synthesis app realizes the function of an image synthesis unit, also known as a compositor, and also the function of an image output unit.

The apps also include a detection app. The detection app detects the gestures of the occupants in the vehicle based on the image signals from the occupant monitor 22, detects finger operations and their line of sight, and detects the facial expressions and gestures of the occupants in the vehicle. It also detects the operator of the touch panel of the operation panel 21.

Of the applications, those that draw image content are assigned a display layer for drawing the image content. These display layers are allocated on the storage unit 6 in a size that allows the necessary image content to be drawn.

Various types of content are stored in the memory unit 6. The content is classified into entertainment content such as movies and television images, meter screens, images captured by the door mirror camera 4a of the electronic mirror 4, images captured by the peripheral cameras 23 (e.g., a front-view camera, a corner camera, a side camera, a rear camera, the door mirror camera 4a, etc.), and around-view image content created by combining images captured by the peripheral cameras 23.

The above-mentioned contents are related to each other, and are registered in a database in the storage unit 6 with a relationship between the main content and the related content. For example, the images captured by the electronic mirror 4 are related to the contents indicating the detection conditions Sa and Sb of the clearance sonar 24a. Furthermore, the images captured by the peripheral cameras 23 mentioned above, i.e., the images captured by the front camera, corner camera, side camera, rear camera, and door mirror camera 4a, are related to each other.

Furthermore, the map image and meter image related to navigation are stored in association with the captured image of the electronic mirror 4. Furthermore, the navigation map image is stored in the storage unit 6 in association with the content of supplementary information. The supplementary information is, for example, the content of the operation instruction button display screen such as the destination, current location, navigation settings, environment settings, map detail display, display change, map display off, destination set, point registration, wide area map change, etc. Furthermore, the map image related to navigation is stored in the storage unit 6 in association with information on facilities such as restaurants close to the map location.

When a certain content stored in the memory unit 6 is regarded as the main content, the control device 11 can read and refer to the related content related to this main content by referring to the memory unit 6. This allows the display processing unit 13 to simultaneously display the main content and the related content on the display 3 or 4b.

The content displayed on each display 3, 4b can be animated. Here, animation refers to a display mode in which the position or size of an image showing the content gradually changes, the image rotates, the user interface moves as a whole in response to a swipe operation, the image gradually fades in or out, or the color of the image changes. However, animation is not limited to these, and any display mode that changes continuously or intermittently from a certain point in time is included in the animation mode.

The control device 11 shown in FIG. 2 executes various applications stored in the memory unit 6. In this embodiment, the control device 11 executes a navigation application stored in the memory unit 6 to realize a navigation function. The control device 11 also executes a detection application stored in the memory unit 6 to detect the presence or absence of occupants seated in the driver's seat (hereinafter, D seat), passenger seat (hereinafter, P seat), or rear seat in the vehicle based on a monitor signal from the occupant monitor 22. It also functions as a gaze detection unit 11b that detects the gaze of an occupant seated in the D seat, P seat, or rear seat.

The control device 11 executes an app stored in the memory unit 6 to realize a function as a determination unit 11c that determines the main content being viewed by the vehicle occupant. The control device 11 executes an app stored in the memory unit 6 to realize a function as a change detection unit 11d that detects a change in the situation related to the main content, a change in the situation of related content related to the main content, or a change in the vehicle occupant viewing the main content. The control device 11 also executes an app stored in the memory unit 6 to realize a function as a driving load determination unit 11e that determines the driving load of the vehicle occupant, particularly the driver.

The display processing unit 13 then changes the display mode of the main content or related content based on the detection result by the change detection unit 11d, and displays the main content and related content within a specified field of view ahead of the vehicle occupant's line of sight.

The above operation will be described with reference to Figure 3 and subsequent figures. As shown in Figure 3, when the power switch is operated to start the vehicle, the control device 11 starts an application such as a navigation application in S1. When the application is started, the vehicle occupant's line of sight is monitored using the occupant monitor 22 in S2, and the display processing unit 13 outputs an image to each display 3, 4b for display.

In S4, the control device 11 determines the content being viewed by the driver and recognizes it as the main content. The driver corresponds to the first vehicle occupant. Here, the content being viewed can be determined by detecting the driver's line of sight and determining which display 3, 4b the driver is looking at.

The control device 11 then determines in S5 whether the situation related to the main content has changed, determines in S6 whether the situation of the related content has changed, and determines in S7 whether the vehicle occupant viewing the main content has changed.

The determination in S5 of "the situation related to the main content has changed" indicates, for example, when the image captured by the electronic mirror 4 is recognized as the main content, and when a change is detected in which an obstacle begins to be recognized in the image captured by the electronic mirror 4 as the vehicle backs up. In addition, it indicates a case where the main content seen by the driver is the detection situation Sa of the clearance sonar 24a, and some kind of change is detected, such as the clearance sonar 24a detecting an obstacle around the vehicle Ca. In other words, when some kind of change occurs in the main content, a YES determination is made in S5.

In addition, the judgment content of S6 "the status of related content has changed" indicates, for example, when the main content being viewed by the driver is the image captured by the electronic mirror 4, there is a change to the detection status Sa of the clearance sonar 24a, which is related content related to the image captured by the electronic mirror 4, such as when a change occurs in which an obstacle begins to be detected.

The judgment in S6, "the status of related content has changed," indicates a case where, when the main content being viewed by the driver is the detection status Sa of the clearance sonar 24a, an obstacle begins to be detected in the captured image of the electronic mirror 4 related to the detection status Sa of the clearance sonar 24a, causing a change in the image recognition status of the captured image. In other words, when any change occurs in the related content related to the main content, a YES judgment is made in S6.

When the control device 11 determines in S5 that the situation related to the main content has changed, or in S6 that the situation of the related content has changed, it changes the display mode in S9 and then identifies the display 3, 4b to be displayed on in S10, and the display processing unit 13 outputs an image to the identified display 3, 4b for display.

The determination in S7 "the passenger viewing the main content has changed" indicates, for example, a case where the driver's line of sight has shifted from the main content and the passenger in the P seat has started viewing the same main content. When the passenger viewing a specific main content has changed, the determination in S7 is YES.
This is equivalent to a vehicle occupant.

When the control device 11 determines in S7 that the vehicle occupant viewing the main content has changed, it detects the line of sight of the second vehicle occupant viewing the main content using the occupant monitor 22 in S8, changes the display mode of the main content or related content in S9, identifies the display (e.g. 3, 4b, etc.) on which to display in S10, and outputs and displays the image in S11.

An example of application of the present invention to actual vehicle operation is described below. For example, when a driver backs up his/her vehicle Ca to park in a parking lot, he/she is looking at the detection status Sa of the clearance sonar 24a displayed on the center display 3 as shown in Figures 4 and 5.

The control device 11 detects that the driver is looking at the detection status Sa of the clearance sonar 24a. At this time, the detection status Sa of the clearance sonar 24a is recognized as the main content. When there are no obstacles around the vehicle Ca and the clearance sonar 24a does not respond to proximity, the display processing unit 13 displays the detection status Sa of the clearance sonar 24a on the center display 3 as shown in FIG. 5.

Thereafter, when the driver reverses the vehicle Ca, assuming that an obstacle Ob is present to the left rear of the vehicle Ca as shown in Figure 6, the clearance sonar 24a detects the obstacle Ob to the left rear of the vehicle Ca.

At this time, the display processing unit 13 displays the detection direction of the obstacle that reacted to the clearance sonar 24a and the image captured by the peripheral camera 23 that captures the surroundings of the detection direction as related content in the field of view. Specifically, the display processing unit 13 displays a warning in the direction in which the obstacle was detected along with the detection status Sa of the clearance sonar 24a, and further displays the image captured by the peripheral camera 23 corresponding to the direction in which the clearance sonar 24a reacted adjacent to the display area of the detection status Sa. In the case shown in FIG. 6, it is preferable to display the image captured by the door mirror camera 4a of the electronic mirror 4 that captures the left rear, as shown in FIG. 7. At this time, the display processing unit 13 preferably displays the image so that it is within the field of view of the vehicle occupant.

As a result, before a situation change is detected by the change detection unit 11d of the control device 11, the display processing unit 13 displays the image captured by the door mirror camera 4a of the electronic mirror 4, which is the related content, on the display 4b of the electronic mirror 4, which is the default predetermined position, and after a situation change is detected, the related content can be moved to the field of view of the center display 3. At this time, the display processing unit 13 may erase the related content that has been displayed on the display 4b of the electronic mirror 4 as the default, or may copy and display it on the center display 3 while leaving it displayed on the display 4b of the electronic mirror 4.
It is desirable to determine in advance which direction (forward, backward, front-side, rear-side, etc.) of the image content captured by the peripheral camera 23 should be displayed for the reaction position of the clearance sonar 24a, and to store the correspondence in advance in the memory unit 6.

It is also desirable for the display processing unit 13 to change the position at which the related content is displayed depending on the relative positional relationship between the position of the vehicle occupant viewing the main content and the main content. For example, when the related content is related to the right side of the driver viewing the main content, it is preferable to display the related content to the right of the main content, and conversely, as shown in Figures 6 and 7, when the related content is related to the left side of the driver viewing the main content, it is preferable to display the related content to the left of the main content. Also, for example, when it is detected that an occupant in the P seat has viewed the content, it may be displayed on the side of the occupant in the P seat. When a dangerous place is shown using a captured image, it is preferable to display the main content at a relative position to the dangerous place with the main content at the center.

Furthermore, even if the driver subsequently moves his/her gaze to the side of the center of a specified center display 3, the control device 11 detects this change in gaze movement, and the display processing unit 13 displays an image captured by the peripheral camera 23 capturing an image in the direction in which the clearance sonar 24a reacts, for the specified center display 3. If the gaze is moved toward the center of the center display 3, an image captured by the backup camera is displayed. The display processing unit 13 displays image content so that it is within the field of view of the destination of the gaze.

Even if the line of sight is not moved after the clearance sonar 24a has finished detecting an obstacle, it is desirable to keep the image captured by the peripheral camera 23 displayed for a certain period of time. This allows the vehicle occupant to check the situation around the vehicle Ca and recognize that there are no problems. This reduces the frequency with which the vehicle occupant changes their line of sight, and reduces the number of times they turn their head.

Consider a case where the driver backs up the vehicle Ca and parks it in a home parking space located at the rear left while checking the display 4b of the left electronic mirror 4, as shown in Figures 8 and 9. In this way, when the driver is looking at the image captured by the left door mirror camera 4a, the control device 11 recognizes the image captured by the electronic mirror 4 as the main content.

In normal driving, not automatic driving, the driver must always check the driving direction, so it is difficult to continue driving while continuously looking at the display 4b of the electronic mirror 4. Also, when passing an oncoming vehicle while driving on a narrow road, or when passing obstacles on the left and right, it is better to drive while paying attention to the left and right electronic mirrors 4 as in the past. For this reason, when the control device 11 detects that the driver is looking at the display 4b of the electronic mirror 4 by the passenger monitor 22, it is better to set the detection situation Sa of the clearance sonar 24a as related content, and the display processing unit 13 may display the detection situation Sa in which the clearance sonar 24a reacts to another obstacle in a part of the display area of the electronic mirror 4, as shown in FIG. 10. As shown in FIG. 10, even if another vehicle Ce is present to the side and rear of the vehicle Ca, the direction can be recognized.

In summary, as shown in FIG. 11, when the display processing unit 13 displays the detection status Sa of the clearance sonar 24a on the center display 3, and further determines that the driver is looking at the detection status Sa of the clearance sonar 24a, the display processing unit 13 displays the image captured by the electronic mirror 4 together with the normal image of the detection status Sa of the clearance sonar 24a on the same display 3.

In addition, when the display processing unit 13 displays the captured image on the display 4b of the electronic mirror 4 and determines that the driver is looking at the electronic mirror 4, the display processing unit 13 displays the detection status Sa of the clearance sonar 24a on the same display 4b along with the normal captured image of the electronic mirror 4. In all other cases, the normal display continues. This makes it possible to draw the driver's attention.

When there are multiple images that the driver should check depending on the driving environment or traffic conditions, the display processing unit 13 may display them together in one place as main content and related content. It is also desirable to display them in a way that is optimized for the location where the occupant is looking.

In addition, it is preferable that the display processing unit 13 performs display processing by changing the related content or changing the display mode of the related content depending on the display priority set for the main content and related content according to the driving environment or traffic conditions. Specifically, the control device 11 sets display priorities for the images captured by the peripheral camera 23 and the images captured by the electronic mirror 4 according to the driving environment or traffic conditions that change from moment to moment, and displays the images on the displays 3 and 4b using the display priorities. At this time, it is preferable to display the images on the displays 3 and 4b in order of the contents with the highest display priorities.

At this time, if the display priority of the related content is changed to be higher than that of the main content, it is recommended to display the related content so that it stands out against the main content. If the situation of the related content changes and its display priority is changed to be higher, it is recommended to display the related content by changing the related content itself or by changing the display mode of the related content.

By detecting the images captured by the surrounding camera 23 and the behavior of the vehicle Ca, the control device 11 can determine the situation in which the vehicle Ca is located in the driving environment or traffic conditions including location and time.

For example, a case will be described in which the position identification unit 11a determines that the vehicle is traveling on an expressway. When the control device 11 detects many other vehicles Ce around the vehicle Ca while traveling on the expressway and detects that the traveling speed of the vehicle Ca has rapidly decreased within a predetermined time, it can determine the driving environment and traffic conditions of the vehicle Ca, such as the vehicle being caught in a traffic jam while traveling on the expressway.

If such driving environment and traffic conditions are stored in a database in the memory unit 6, the control device 11 can refer to the database in the memory unit 6 and detect using the surrounding camera 23 and distance detection sensor 24 to determine the degree to which other vehicles Ce are concentrated around the host vehicle Ca, or whether the number of other vehicles Ce is increasing around the host vehicle Ca, thereby predicting whether or not traffic congestion will occur, or the extent of the traffic congestion.

For this reason, when the control device 11 determines that the vehicle has decelerated more rapidly than a predetermined amount while traveling on a highway, it can detect that a traffic jam has occurred by referring to the database in the memory unit 6, and can further predict the extent of the traffic jam. For example, it can display an image captured by a backup camera, which is one of the surrounding cameras 23 and shows the situation of the vehicle following the vehicle Ca, as related content on the center display 3, which displays a map screen as the main content. By performing display processing in this manner, it is possible to alert the driver.

As another example, a case will be described where the location identification unit 11a determines that the vehicle is traveling near the driver's home address. When it is determined that the vehicle is traveling near the driver's home address, the control unit 11 determines whether the shift range is in the R range and, by referring to the database stored in the memory unit 6, estimates events that are likely to cause an accident when parking in the home parking space, and reflects this in the display priority of the image to be displayed on the display 3 or 4b.

For example, when the driver reverses the vehicle Ca while turning the steering wheel to the left, the driver reverses the vehicle Ca while checking the image on the display 4b of the left electronic mirror 4, as shown in Figure 10. However, if the driver pays too much attention to the display 4b of the left electronic mirror 4, it is easy to overlook another obstacle (not shown) on the right side. If the accident event is registered in the database of the memory unit 6, the control device 11 can refer to the accident event.

When an obstacle is detected to the right rear by the clearance sonar 24a, the display processing unit 13 may display the right detection situation Sb on the display 4b of the electronic mirror 4 instead of the left detection situation Sa displayed as related content, as shown in Fig. 12. Also, as shown in Fig. 12, the right detection situation Sb may be displayed larger than the detection situation Sa to alert the driver. In the example shown in Fig. 12, only the right detection situation Sb is displayed as related content, but the left detection situation Sa may also be displayed together.

According to this embodiment, the display mode of the main content or related content is changed based on the detection result by the change detection unit 11d, and the main content and related content are displayed in the field of view of the vehicle occupant who is viewing the content. This allows the vehicle occupant to simultaneously view the main content and the related content. This allows the vehicle occupant to view appropriate information while minimizing the number of times they move their line of sight or shake their head.

Second Embodiment
The second embodiment will be described with reference to Fig. 13 to Fig. 16. In the second embodiment, the relationship between the main content and the related content when the occupant views a map screen drawn according to a navigation application will be described.

The map screen B1 shown in Figure 13 is drawn on the display 3 etc. when the navigation app is executed. Under normal circumstances, it is preferable for the display processing unit 13 to display the map screen B1 large and filling the display screen of the center display 3. This is because the driver can easily check the map screen B1 even if he or she glances at it while paying attention to the road ahead during normal driving. In other words, when the control unit 11 detects via the occupant monitor 22 that the driver is looking at the map screen B1, it is preferable for the display processing unit 13 to display the map screen B1 large and filling the display screen of the center display 3 so that no unnecessary content is displayed.

In addition, since it would be inconvenient if the necessary information was not displayed without the occupant's operation, it is also possible to display the necessary information in advance, such as map direction information B2 and VICS (registered trademark) information acquisition time B3, as shown in Figure 14.

In addition, when the control device 11 detects that the map screen B1 has been viewed by the occupant for a certain period of time, the display processing unit 13 may display supplementary information for the map screen B1 as related content, as shown in Fig. 15. The supplementary information shown in Fig. 15 is, for example, the content of the operation instruction button display screen B8, such as destination setting B4, current location setting B5, other navigation settings B6, environment setting B7, map detail display, display change, map display off, destination setting, point registration, and wide-area map change.

In other words, if the control device 11 determines that the driver has not looked at the screen or has looked at it for less than a certain period of time, the display processing unit 13 does not display the supplementary information, but if the driver continues to look at the screen for a certain period of time, the supplementary information is superimposed. This allows information to be provided while gradually increasing the amount of information, and information appropriate to the situation can be provided. The driver will be able to intuitively confirm more information with a delay, which reduces the cognitive processing burden on the driver.

Furthermore, the control device 11 and the display processing unit 13 may change the content displayed on the center display 3 depending on whether it is being viewed by the driver or by a passenger in seat P. That is, when the control device 11 determines in S7 of Fig. 3 that the passenger viewing the map screen has changed and that the passenger in seat P is now viewing the navigation map information, the display processing unit 13 may, after detecting the line of sight of the passenger in seat P in S8, display, as related content, entertainment information being held near the current location displayed on the map screen B1 or, as shown in Fig. 16, recommended stores in the vicinity B9 in S9 to S11.

At this time, the display processing unit 13 may display the related content on the side of the passenger in seat P who is looking at the map screen B1. At this time, it may be displayed within the visual field of the passenger in seat P. This allows the passenger in seat P to reduce the frequency with which he or she moves his or her line of sight, and therefore the number of times he or she turns his or her head.

In addition, the control device 11 preferably determines the driver's driving load using the driving load determination unit 11e, and the display processing unit 13 changes the amount of information of the related content according to the driver's driving load and processes the display. The driving load can be calculated by detecting the driving actuator operated by the driver, the operation status of the navigation display screen, the autonomous driving level, or the state of the occupant using the occupant monitor 22 using known technology.

For example, when the control device 11 determines that the driver's driving load is within the standard range using the driving load determination unit 11e, the amount of information is displayed as a normal predetermined amount, but when the control device 11 determines that the driving load is lower than the standard, the amount of information may be increased. Conversely, when the control device 11 determines that the driver's driving load is higher than the standard, it is preferable to use a less noticeable expression than the standard or to reduce the amount of information. In other words, for related content, it is preferable to change the amount of information depending on the driver's driving load and display it.

For example, when the vehicle is stopped or when the navigation destination or current location is being set while the vehicle is stopped, the driving load is low, so the amount of information on related content may be displayed in a larger amount than standard. Also, during automated driving at level III or higher, the driving load on the driver may be determined to be low, and similarly the amount of information displayed may be larger than standard. Also, during following driving using ACC or steady driving using lane keep assist (LAS), the driving load may be determined to be low, and similarly the amount of information on related content may be displayed in a larger amount than standard.

Conversely, if the control device 11 determines using the perimeter monitoring sensor that there are more obstacles around the vehicle than a predetermined amount, or determines by referring to traffic information from VICS (registered trademark) or the like that the vehicle is traveling on a highway where the traffic volume in the vicinity is greater than a predetermined amount, or determines using the perimeter monitoring sensor that another vehicle Ce is entering the vehicle's lane while traveling on a highway, or determines that people are present or moving around the vehicle even while traveling on a general road, it may determine that the driving load is high and display the amount of information in the related content less than standard or inconspicuously.

Furthermore, when lane change assist is being performed in level II driving assistance, the driver is legally obligated to monitor, especially while steering the vehicle. The lane change assist is called Lane Change Assist. The control device 11 may determine that the driving load is high while lane change assist is being performed, and may reduce the amount of information on related content below standard or display it in an inconspicuous manner.

It is desirable for the display processing unit 13 to determine whether or not to display related content or change the display mode depending on whether or not the related content is being viewed by one or more vehicle occupants. This is because safety can be improved by preventing vehicle occupants such as the driver from viewing the related content depending on the situation.

For example, if the display processing unit 13 displays video content that is useful for the passenger in the P seat as related content, it will be an unsafe distraction for the driver. It is advisable to stop displaying the video content when it is detected that the driver has viewed the related content. In addition, when there are multiple passengers viewing the video content, it is desirable for the control unit 11 to determine which passenger is viewing it, and to stop displaying the video content if the driver is viewing it and there is a problem with the safety of vehicle operation.

When the vehicle changes from being stopped to being driven, it is desirable to stop the video content if it is detected that the driver is watching entertainment video content. However, during autonomous driving at a certain level or higher, for example during autonomous driving at level III or higher, the driver may be treated the same as a passenger in the P seat, and the display processing unit 13 may continue to display the video content.

Other Embodiments
The present disclosure is not limited to the above-described embodiment, and can be implemented in various modifications and variations without departing from the spirit and scope of the present disclosure. For example, the following modifications or extensions are possible.

When the change detection conditions are met by the change detection unit 11d, the main content and related content are displayed in the field of view ahead of the vehicle occupant's line of sight, but conversely, it is also possible to adjust the width of the visible range using a polarizing filter or the like and display the content in the field of view.

In the drawing, 11c indicates a judgment unit, 11d indicates a change detection unit, 11e indicates a driving load judgment unit, and 13 indicates a display processing unit.

The control device 11 and the display processing unit 13 described in the present disclosure may be realized by a dedicated computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, the control device 11, the display processing unit 13, and the method described in the present disclosure may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control device 11, the display processing unit 13, and the method described in the present disclosure may be realized by one or more dedicated computers configured by combining a processor and a memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. In addition, the computer program may be stored in a computer-readable non-transient tangible recording medium as instructions executed by the computer.

Although the present disclosure has been described in accordance with the above-described embodiment, it is understood that the present disclosure is not limited to the embodiment or structure. The present disclosure also encompasses various modifications and modifications within the scope of equivalents. In addition, various combinations and forms, as well as other combinations and forms including one, more, or less than one element, are within the scope and concept of the present disclosure.

Claims (7)

a determination unit (11c) for determining whether a main content being viewed by a vehicle occupant is a map screen ;
a change detection unit (11d) that determines that the traveling speed of the vehicle has suddenly decreased within a predetermined time from a speed signal of the vehicle detected by a vehicle speed sensor, and determines that the number of other vehicles has increased around the vehicle by detecting a change in a situation in an image captured by a backup camera as a related content related to the map screen;
a display processing unit (13) that changes the display mode of the image captured by the rear camera based on the detection result by the change detection unit, and displays the map screen and the image captured by the rear camera in the field of view ahead of the line of sight of the vehicle occupant. 2. The vehicle display system of claim 1, wherein before a situation change is detected by the change detection unit, the display processing unit displays the related content at a default predetermined position, and after the situation change is detected, the display processing unit moves or copies the related content into the field of view. 3. The vehicular display system according to claim 1, wherein the display processing unit changes a position where the related content is displayed depending on a relative positional relationship between a position of the vehicle occupant viewing the primary content and the primary content. 3. The vehicle display system according to claim 1, wherein the display processing unit changes the related content or changes the display mode of the related content using display priorities set for the main content and the related content depending on the driving environment or traffic conditions. The vehicular display system according to claim 1 or 2 , wherein the display processing unit determines whether to display or changes a display mode depending on whether or not one or more of the vehicle occupants are checking the related content. A determination unit (11c) determines whether the main content being viewed by the vehicle occupant is a map screen ;
A change detection unit (11d) determines that the traveling speed of the vehicle has rapidly decreased within a predetermined time from a speed signal of the vehicle detected by a vehicle speed sensor, and determines that the number of other vehicles has increased around the vehicle by detecting a change in the situation of an image captured by a backup camera as related content related to the map screen;
A display processing unit (13) changes the display mode of the image captured by the rear camera based on the detection result by the change detection unit, and displays the map screen and the image captured by the rear camera within the field of view of the vehicle occupant. a step of causing a determination unit (11c) to determine whether a main content being viewed by a vehicle occupant is a map screen ;
a step of causing a change detection unit (11d) to determine that the traveling speed of the vehicle has rapidly decreased within a predetermined time from a speed signal of the vehicle detected by a vehicle speed sensor, and to determine that the number of other vehicles has increased around the vehicle by detecting a change in the situation of an image captured by a backup camera as related content related to the map screen ;
A display program for a vehicle, comprising a procedure in which a display processing unit (13) changes the display mode of the image captured by the rear camera based on the detection result by the change detection unit , and displays the map screen and the image captured by the rear camera within the field of view of the vehicle occupant's line of sight.

Images