WO2021258672A1 - 基于车载数字人的交互方法及装置、存储介质 - Google Patents

基于车载数字人的交互方法及装置、存储介质 Download PDF

Info

Publication number
WO2021258672A1
WO2021258672A1 PCT/CN2020/136256 CN2020136256W WO2021258672A1 WO 2021258672 A1 WO2021258672 A1 WO 2021258672A1 CN 2020136256 W CN2020136256 W CN 2020136256W WO 2021258672 A1 WO2021258672 A1 WO 2021258672A1
Authority
WO
WIPO (PCT)
Prior art keywords
vehicle
relative position
getting
position information
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/136256
Other languages
English (en)
French (fr)
Inventor
周群艳
曾彬
李轲
吴阳平
许亮
许亲亲
林楠
王玉生
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Sensetime Lingang Intelligent Technology Co Ltd
Original Assignee
Shanghai Sensetime Lingang Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Sensetime Lingang Intelligent Technology Co Ltd filed Critical Shanghai Sensetime Lingang Intelligent Technology Co Ltd
Priority to EP20941609.8A priority Critical patent/EP4170466A4/en
Priority to JP2022524647A priority patent/JP2022553773A/ja
Priority to KR1020217034787A priority patent/KR20220000903A/ko
Publication of WO2021258672A1 publication Critical patent/WO2021258672A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—Three-dimensional [3D] animation
    • G06T13/40—Three-dimensional [3D] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/80—Two-dimensional [2D] animation, e.g. using sprites
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2213/00—Indexing scheme for animation
    • G06T2213/12—Rule based animation

Definitions

  • the present disclosure relates to the field of augmented reality, and in particular to an interaction method and device based on a vehicle-mounted digital human, and a storage medium.
  • robots can be placed in the car, and after a person enters the car, the robot can interact with the person in the car.
  • the interaction mode between the robot and the people in the car is relatively fixed and lacks civilization.
  • the present disclosure provides an interaction method, device, and storage medium based on a vehicle-mounted digital human.
  • a vehicle-mounted digital human-based interaction method comprising: acquiring a video stream of an area in the vehicle; and detecting relative position information of objects getting on and off the vehicle based on the images in the video stream
  • the relative position information includes: the first relative position information of the object getting on and off the vehicle in the image or the second relative position information of the object getting on and off the vehicle and the vehicle; according to the relative position information, generating and The animation of the digital person displayed in the display device provided on the vehicle performing the corresponding steering is displayed.
  • the detecting the relative position information of the object getting on and off the vehicle according to the image in the video stream includes: determining the location of at least one part of the object getting on and off the vehicle in the image coordinate system corresponding to the image The image coordinate area of the image; use the image coordinate area as the first relative position information of the object getting on and off the vehicle in the image.
  • the detecting the relative position information of the object getting on and off the vehicle according to the image in the video stream includes: determining where at least one part of the object getting on and off the vehicle is located in the image coordinate system corresponding to the image The image coordinate area; according to the mapping relationship between the image coordinates and the interior space coordinates, determine the interior space coordinate region corresponding to the image coordinate region where at least one part of the object getting on and off the vehicle is located; convert the interior space coordinates The area is used as the second relative position information of the object of getting on and off the vehicle and the vehicle.
  • generating and displaying the animation of the digital person displayed on the display device provided on the car to perform the corresponding turning includes: according to the position information in the pre-established image and the digital person The mapping relationship between the orientations of the vehicle is determined, and the target orientation corresponding to the relative position information is determined; an animation of the digital person displayed on the display device provided on the vehicle turning to the target orientation is generated and displayed.
  • generating and displaying the animation of the digital person displayed on the display device provided on the vehicle performing the corresponding steering includes: responding to the detected relative position of the object getting on and off the vehicle
  • the information includes multiple relative positions of multiple objects getting on and off the vehicle, and determining the relative position of the target among the multiple relative positions; according to the relative position of the target, generating and displaying the digital person displayed on the display device provided on the vehicle Perform the animation of the corresponding turn.
  • the detected relative position information of the objects getting on and off the vehicle in response to the detection includes multiple relative positions of multiple objects getting on and off the vehicle
  • determining the relative position of the target in the multiple relative positions includes: responding
  • the detected relative position information of the objects getting on and off the vehicle includes multiple relative positions of the multiple objects on the The position is determined as the relative position of the target.
  • the priority order of the different location information includes at least one of the following: the priority of the location information corresponding to the driving area is higher than the priority of the location information corresponding to the co-pilot area; the location corresponding to the front row area The priority of the information is higher than the priority of the position information corresponding to the rear area; the priority of the position information corresponding to the dedicated seat is higher than the priority of the position information corresponding to the non-dedicated seat.
  • the method before detecting the relative position information of the object getting on and off the vehicle based on the image in the video stream, the method further includes: determining the upper and lower position of the object based on the multi-frame images with a time-series relationship in the video stream.
  • Car intention according to the object's intention to get on and off the car, determine the object of getting on and off the car.
  • the determining the object’s intent to get on and off the vehicle based on the multi-frame images with a time-series relationship in the video stream includes at least one of the following: in response to the multi-frame image being less than a preset distance from the door At least one part of the object is detected within the range of the distance, and the intention of the object to get off the vehicle is determined; in response to the multiple frames of images where the distance from the door of the vehicle is less than the preset distance, no part of the object is detected and the door is located Open state, determine the subject's intention to get on the vehicle.
  • the method further includes: detecting the attribute information of the object getting on and off the vehicle based on the image in the video stream; generating and displaying the digital person according to the attribute information of the object getting on and off the vehicle Broadcast or display the animation of the multimedia information corresponding to the attribute information.
  • the object to get on and off the vehicle includes at least one of the following: people and pets;
  • the attribute information of the object includes at least one of the following: identity information, gender information, and category information;
  • the multimedia information includes at least the following One: expression information, action information, and voice information.
  • an interactive device based on a vehicle-mounted digital human.
  • the device includes: an acquisition module for acquiring a video stream of an area in the vehicle; a relative position detection module for The image detects the relative position information of the object getting on and off the car, the relative position information includes: the first relative position information of the object getting on and off the car in the image or the second relative position of the object getting on and off the car and the car Position information; the first interaction module is used to generate and display the animation of the digital person displayed in the display device provided on the vehicle performing the corresponding steering according to the relative position information.
  • the relative position detection module includes: a first determining sub-module, configured to determine, in an image coordinate system corresponding to the image, an image coordinate area where at least one part of the object getting on and off the vehicle is located;
  • the second determining sub-module is configured to use the image coordinate area as the first relative position information of the object getting on and off the vehicle in the image.
  • the relative position detection module includes: a third determining sub-module, configured to determine, in the image coordinate system corresponding to the image, the image coordinate area where at least one part of the object getting on and off the vehicle is located;
  • the fourth determining sub-module is used to determine the in-vehicle space coordinate area corresponding to the image coordinate area where at least one part of the object getting on and off the vehicle is located according to the mapping relationship between the image coordinates and the in-vehicle space coordinates;
  • the fifth determining sub-module The module is configured to use the space coordinate area in the vehicle as the second relative position information of the object of getting on and off the vehicle and the vehicle.
  • the first interaction module includes: an orientation determination sub-module, configured to determine a target corresponding to the relative position information according to a pre-established mapping relationship between the position information in the image and the orientation of the digital person Orientation; a first interaction sub-module for generating and displaying the animation of the digital person turning to the target orientation displayed in the display device set on the vehicle.
  • the first interaction module includes: a sixth determining sub-module, configured to determine that the detected relative position information of objects getting on and off the vehicle includes multiple relative positions of multiple objects getting on and off the vehicle. The relative position of the target in the multiple relative positions; the second interaction sub-module is used to generate and display the animation of the digital person displayed in the display device provided on the vehicle performing the corresponding steering according to the relative position of the target.
  • the sixth determining sub-module includes: a determining unit, configured to respond to the detected relative position information of objects getting on and off the vehicle, including multiple relative positions of multiple objects getting on and off the vehicle, according to a predetermined The priority order of the different position information, the relative position with the highest priority among the multiple relative positions is determined as the target relative position.
  • the priority order of the different location information includes at least one of the following: the priority of the location information corresponding to the driving area is higher than the priority of the location information corresponding to the co-pilot area; the location corresponding to the front row area The priority of the information is higher than the priority of the position information corresponding to the rear area; the priority of the position information corresponding to the dedicated seat is higher than the priority of the position information corresponding to the non-dedicated seat.
  • the device further includes: a boarding and boarding intention determination module, configured to determine the object’s boarding and boarding intention based on the multiple frames of images with time-series relationships in the video stream; and a boarding and boarding object determination module, configured to determine The getting on and off intention of the object determines the getting on and off object.
  • a boarding and boarding intention determination module configured to determine the object’s boarding and boarding intention based on the multiple frames of images with time-series relationships in the video stream
  • a boarding and boarding object determination module configured to determine The getting on and off intention of the object determines the getting on and off object.
  • the getting on and off intention determination module includes at least one of the following: a seventh determination sub-module, configured to respond to the detection of an object within a range that is less than a preset distance from the vehicle door in the multi-frame image At least one part determines the object’s intention to get off the car; the eighth determining sub-module is used to respond to the fact that no part of the object is detected within a range of less than a preset distance from the car door in the multi-frame images and the car door is in Open state, determine the subject's intention to get on the vehicle.
  • the device further includes: an attribute information detection module, configured to detect the attribute information of the object getting on and off the vehicle based on the image in the video stream;
  • the attribute information of the object of the vehicle generates and displays the animation of the digital person broadcasting or displaying the multimedia information corresponding to the attribute information.
  • the object to get on and off the vehicle includes at least one of the following: people and pets;
  • the attribute information of the object includes at least one of the following: identity information, gender information, and category information;
  • the multimedia information includes at least the following One: expression information, action information, and voice information.
  • a computer-readable storage medium stores a computer program, and when a processor executes the computer program, the processor is configured to execute the above-mentioned first The interactive method based on a vehicle-mounted digital human as described in any embodiment of the aspect.
  • an interactive device based on a vehicle-mounted digital human including: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to call the When the executable instructions stored in the memory are used, the vehicle-mounted digital human-based interaction method described in any one of the embodiments of the first aspect is implemented.
  • a computer program product including computer-readable code.
  • the processor in the device executes the method for implementing the method provided in any of the above embodiments. Instructions based on the interactive method of the vehicle-mounted digital human.
  • the relative position information of the objects getting on and off the car is detected by analyzing the images in the video stream of the area inside the car, and based on the relative position information, the information displayed on the display device set on the car is generated and displayed
  • the digital person executes the animation of the corresponding turning.
  • an animation of the digital person performing the corresponding steering is generated and displayed, making the human-computer interaction method more in line with human interaction habits and the interaction process is more natural , Let people feel the warmth of human-computer interaction, and enhance the ride pleasure, comfort and sense of accompany.
  • Fig. 1 is a flow chart showing a method for interaction based on a vehicle-mounted digital human according to an exemplary embodiment of the present disclosure
  • Fig. 2 is a flowchart of step 102 according to an exemplary embodiment of the present disclosure
  • Fig. 3 is a flowchart showing step 102 according to another exemplary embodiment of the present disclosure.
  • Fig. 4 is a flowchart of step 103 according to an exemplary embodiment of the present disclosure.
  • Fig. 5 is a flowchart showing step 103 according to another exemplary embodiment of the present disclosure.
  • Fig. 6 is a flow chart showing a method for interaction based on a vehicle-mounted digital human according to another exemplary embodiment of the present disclosure
  • Fig. 7 is a flowchart of step 104 according to an exemplary embodiment of the present disclosure.
  • Fig. 8 is a flowchart of a vehicle-mounted digital human-based interaction method according to still another exemplary embodiment of the present disclosure.
  • Fig. 9 is a schematic diagram showing a vehicle-mounted digital person according to an exemplary embodiment of the present disclosure.
  • Fig. 10 is a block diagram showing a vehicle-mounted digital human interaction device according to an exemplary embodiment of the present disclosure
  • Fig. 11 is a schematic diagram showing the hardware structure of an interactive device based on a vehicle-mounted digital human according to an exemplary embodiment of the present disclosure.
  • first, second, third, etc. may be used in this disclosure to describe various information, the information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other.
  • first information may also be referred to as second information, and similarly, the second information may also be referred to as first information.
  • word “if” as used herein can be interpreted as "when” or “when” or “in response to certainty.”
  • the embodiments of the present disclosure provide a vehicle-mounted digital human-based interaction method, which can be used for drivable machinery and equipment, such as smart vehicles, smart cabins that simulate vehicle driving, and the like.
  • Fig. 1 shows a vehicle-mounted digital human interaction method according to an exemplary embodiment, which includes the following steps:
  • step 101 the video stream of the in-vehicle area is acquired.
  • the video stream of the area in the vehicle can be obtained through the vehicle-mounted camera.
  • the vehicle-mounted camera can be set on the center console, on the front windshield glass, on the rearview mirror of the vehicle, or any other location that can photograph the area inside the vehicle.
  • the in-vehicle area includes at least one of the following: a driving area, a co-pilot area, and a rear row area.
  • step 102 the relative position information of the object getting on and off the vehicle is detected based on the image in the video stream.
  • the objects getting on and off the vehicle include at least one of the following: people and pets.
  • the relative position information includes the first relative position information or the second relative position information, where the first relative position information is the relative position information of the object getting on and off the vehicle in the image of the video stream, and the image coordinates of the object in the image coordinate system can be used.
  • the second relative position information is the relative position information of the object getting on and off the car and the car, and it can be represented by the space coordinate area of the object getting on and off the car in the car interior space coordinate system.
  • step 103 according to the relative position information, an animation of the digital person displayed in the display device provided on the vehicle performing a corresponding turning is generated and displayed.
  • the digital person is a virtual image generated by software, and can be displayed on a display device installed in a car, where the display device may include, but is not limited to, a central control display screen or a vehicle-mounted tablet device.
  • the relative position information can be used to generate and display the animation of the digital person displayed on the display device to perform the corresponding turning, for example, to generate and display the digital person from the previous orientation to get on and off the car The animation of the orientation of the object.
  • the relative position information of the objects getting on and off the car is detected by analyzing the image in the video stream of the area inside the car, and based on the relative position information, the number displayed on the display device set on the car is generated and displayed
  • the person executes the animation of the corresponding turn.
  • an animation of the digital person performing the corresponding steering is generated and displayed, making the human-computer interaction method more in line with human interaction habits and the interaction process is more natural , Let people feel the warmth of human-computer interaction, and enhance the ride pleasure, comfort and sense of accompany.
  • the interactive method based on a digital person in a vehicle provided by the embodiment of the present disclosure does not require the person to input any additional language or action to accurately determine the direction of the coming person and realize the real Senseless recognition and interaction.
  • step 102 may include:
  • step 201 in the image coordinate system corresponding to the image, the image coordinate area where at least one part of the object getting on and off the vehicle is located is determined.
  • the image coordinate system is a two-dimensional coordinate system corresponding to the image in the video stream.
  • a pre-trained Region Proposal Network RPN
  • RPN Region Proposal Network
  • At least one part can include but is not limited to face, hand, Torso and so on.
  • the image coordinate area where at least one part of the object getting on and off the vehicle is located can be determined.
  • step 202 the image coordinate area is used as the first relative position information of the object of getting on and off the vehicle in the image.
  • the image coordinate area where at least one part of the object getting on and off is located is the image coordinate area 1 in the image coordinate system, then the image coordinate area 1 is taken as the first relative position of the object on getting on and off in the image. information.
  • the image coordinate area where at least one part of the object getting on and off the vehicle is located can be determined, and the determined image coordinate area is used as the first relative position for subsequent reference
  • the first relative position generates and displays the animation of the digital person performing the corresponding steering, which is highly usable.
  • step 102 may include:
  • step 301 in the image coordinate system corresponding to the image, determine the image coordinate area where at least one part of the object getting on and off the vehicle is located.
  • the image coordinate area where at least one part of the object getting on and off the vehicle is located can be determined in the same manner as in step 201, which will not be repeated here.
  • step 302 according to the mapping relationship between the image coordinates and the in-vehicle space coordinates, the in-vehicle space coordinate area corresponding to the image coordinate area where at least one part of the object getting on and off the vehicle is located is determined.
  • the in-vehicle space coordinate area is a coordinate area in the in-vehicle space coordinate system
  • the in-vehicle space coordinate system is a three-dimensional space coordinate system covering the in-vehicle area.
  • the mapping relationship between the image coordinates and the space coordinates in the vehicle is predetermined. After the image coordinate area where at least one part of the object getting on and off the vehicle is located is determined, the up and down can be determined according to the above mapping relationship. The space coordinate area in the car corresponding to the object of the car.
  • step 303 the space coordinate area in the vehicle is used as the second relative position information of the object of getting on and off the vehicle and the vehicle.
  • the image coordinate area where at least one part of the object getting on and off the vehicle is located is the image coordinate area 1 in the image coordinate system.
  • the in-vehicle space coordinate area corresponding to the image coordinate area 1 is the driving area, then the driving area As the second relative position information of the object of getting on and off the car and the car.
  • the interior space corresponding to the object getting on and off the vehicle can be determined according to the mapping relationship between the image coordinates and the interior space coordinates of the vehicle.
  • the space coordinate area in the car is used as the second relative position of the object getting on and off the car and the car, so that the animation of the digital person performing the corresponding steering is generated and displayed according to the second relative position, which is highly usable.
  • step 103 may include:
  • a target orientation corresponding to the relative position information is determined according to a pre-established mapping relationship between the position information in the image and the orientation of the digital person.
  • the mapping relationship between the position information in the image and the orientation of the digital person is established in advance. For example, the orientation of the digital person corresponding to the position information belonging to area position 1 in the image is southwest facing, which belongs to the area position. The orientation of the digital person corresponding to the position information of 2 is southeast orientation, etc. According to this mapping relationship, after the relative position information is determined, the corresponding target orientation can be determined.
  • step 402 an animation of the digital person turning to the target direction displayed on the display device provided on the vehicle is generated and displayed.
  • the digital person has been displayed on the display device, and according to the determined target orientation, an animation for turning the digital person toward the target orientation can be generated and displayed.
  • an animation for turning the digital person toward the target orientation can be generated and displayed.
  • the digital person was facing the direction of due south before, and the target direction was the southwest direction, and the animation generated and displayed was the animation of the digital person turning from the direction of due south to the southwest direction.
  • the target orientation corresponding to the relative position information of the getting on and off objects can be determined, and an animation of the digital person turning to the target orientation can be generated and displayed. Therefore, in the case of an object getting on and off the car, the digital person is turned to the corresponding target orientation, making the way of human-computer interaction more in line with human interaction habits, and the interaction process is more natural, making people feel the warmth of human-computer interaction, and improving the ride Fun, comfort and sense of care.
  • step 103 may include:
  • step 501 in response to the detected relative position information of the object getting on and off the vehicle including information representing multiple relative positions of the multiple objects getting on and off the vehicle, the relative position of the target among the multiple relative positions is determined.
  • the detected relative position information of the objects getting on and off the car includes information that characterizes multiple relative positions, for example, multiple objects get on or get off the car at the same time, or some objects get on the car at the same time , Some objects get off the car, you can determine the relative position of the target among multiple relative positions.
  • step 502 according to the relative position of the target, an animation of the digital person displayed on the display device provided on the vehicle performing a corresponding turning is generated and displayed.
  • an animation of the digital person displayed on the display device provided on the vehicle performing the corresponding steering is generated and displayed according to the relative position of the target among the multiple relative positions.
  • an animation of the digital person turning to the target direction is generated and displayed, and the target direction is the direction corresponding to the relative position of the target.
  • step 501 includes:
  • the priority order of different location information includes at least one of the following: the priority of the location information corresponding to the driving area is higher than the priority of the location information corresponding to the co-pilot area; the location information corresponding to the front row area The priority is higher than the priority of the location information corresponding to the rear area; the priority of the location information corresponding to the dedicated seat is higher than the priority of the location information corresponding to the non-dedicated seat.
  • the priority of the location information of the driving area is higher than the priority of the location information of other areas. class. If the information representing the multiple relative positions of multiple objects getting on and off the vehicle does not include the position information of the driving area, but includes the position information of the passenger area and the rear area, the priority of the position information of the passenger area is higher than that of the rear area. Prioritize the location information of the area.
  • Special seats can include but are not limited to child seats, special seats for other people, etc. If special seats are installed in the car, it can be considered that the priority of the position information of the special seats is higher than that of the corresponding non-dedicated seats The priority of the information.
  • the relative position with the highest priority among multiple relative positions can be determined as the target relative position, and then based on this target relative position, the digital person displayed in the display device set on the car is generated and displayed to execute the corresponding Animation of turning.
  • the detected relative position information of the objects getting on and off the vehicle includes information representing multiple relative positions of the objects on the The relative position with the highest priority among the multiple relative positions is determined as the target relative position, so that the target orientation corresponding to the digital person is determined according to the target relative position, and the animation of the digital person turning to the target direction is generated and displayed, so as to get on and off the object
  • the digital person turn to the object with high priority, so that the way of human-computer interaction is more in line with human interaction habits, and the interaction process is more natural, making people feel the warmth of human-computer interaction, and enhancing the fun of riding a car. Comfort and sense of companionship.
  • the method may further include:
  • step 104 based on the multiple frames of images with a time-series relationship in the video stream, the object's intention to get on and off the vehicle is determined.
  • the multi-frame images having a time series relationship may be multi-frame images that are adjacent in time series.
  • step 105 the object of getting on and off the vehicle is determined according to the intention of getting on and off the vehicle of the object.
  • the object is the object of getting on the vehicle according to the intention of getting on the vehicle of the object
  • the object of getting off the vehicle can be determined according to the intention of getting off the vehicle of the object
  • step 104 may include at least one of the following:
  • step 601 in response to detecting at least one part of the object within a range of the multiple frames of images where the distance from the vehicle door is less than a preset distance, the object's intention to get off the vehicle is determined.
  • the distance from any door of the multi-frame image is less than the preset distance, and at least one part of the object is detected, such as the hand, the head, the leg, etc., it can be determined
  • the subject's intention to get off the vehicle For example, first detect the pixel area corresponding to any car door, and within a preset distance from this pixel area, detect the hand or head or leg of an object, and then determine the object's intention to get off the car .
  • step 602 in response to that no part of the object is detected within a range where the distance from the door of the vehicle is less than a preset distance in the multi-frame images and the door of the vehicle is in an open state, the intention of the object to get on the vehicle is determined.
  • the pixel area corresponding to any one of the car doors is detected first, and within the preset distance from this pixel area, no part of the object is detected, indicating that there is no object currently Get on the vehicle through the door, and at the same time determine that the door is in an open state based on multiple frames of images, then it can be considered that an object wants to enter the vehicle by opening the door, and the object's intention to get on the vehicle can be determined at this time.
  • the detection results of sensors on the vehicle may be combined to further determine the object's intention to get on and off the vehicle.
  • Sensors include, but are not limited to, door sensors, pressure sensors on seats, etc.
  • the door sensor shows that the door is unlocked
  • the pressure sensor installed on the seat shows that the object is on the seat.
  • the door sensor shows that the door is unlocked, and the pressure sensor installed on the seat shows that there is no object on the seat. At this time, if there is no object in the multi-frame image that is less than the preset distance from the door. If any part of the object is detected and the door is in an open state, then the object's intention to board the vehicle can be determined.
  • multiple frames of images with timing relationships in the video stream can be combined to determine the object's intent to get on and off the vehicle, thereby determining the object of getting on and off the vehicle, so as to subsequently generate and display the animation of the digital person performing the steering, which is highly usable.
  • the foregoing method may further include:
  • step 106 the attribute information of the object getting on and off the vehicle is detected based on the image in the video stream.
  • the facial features of the objects getting on and off the vehicle are extracted, and the extracted facial features are compared with the pre-stored facial features to determine the ratio
  • the attribute information of the object getting on and off the vehicle can be determined.
  • the attribute information of the object includes at least one of the following: identity information, gender information, and category information.
  • identity information is used to indicate the specific identities of the objects getting on and off the bus, such as Zhang San and Li Si
  • the gender information is used to indicate the gender of the objects getting on and off the bus, such as male or female
  • the type information is used to indicate the type of the objects getting on and off the bus. , Such as whether the type is an adult, a child, or a pet.
  • step 107 according to the attribute information of the object getting on and off the vehicle, an animation of the digital person broadcasting or displaying the multimedia information corresponding to the attribute information is generated and displayed.
  • the multimedia information includes at least one of the following: expression information, action information, and voice information.
  • the attribute information of the object getting on and off the car includes the identity information of Zhang San, the gender information of female, and the type information of adult, then a digital person corresponding to the identity information can be generated and displayed on the display device, as shown in Figure 9 As shown, further, an animation in which the digital person broadcasts or displays at least one of an expression, an action, and a voice corresponding to the attribute information is generated and displayed.
  • the digital person can chat with the people in the car, and say hello to the people in the car through an animation including at least one of voice, expression, and action, "Good morning, Zhang San”.
  • At least one of the gender information and category information of the person can be combined to broadcast or show animation to say hello, such as "good morning, lady".
  • the digital person can automatically ask whether it is necessary to introduce the car condition and other information through an animation. If the person in the car answers "yes”, the digital person can introduce the car condition, if the person in the car answers or "no", then the digital person It can enter the standby state, and the digital person can indicate that he has entered the standby state through animation of at least one of voice, action, and expression.
  • the animation of the digital person broadcasting or displaying the multimedia information corresponding to the attribute information is generated and displayed, so that the interaction between the digital person and the person in the vehicle is more multi-modal and improves The degree of intelligence of the digital person.
  • multiple objects get off the car at the same time, and the relative position with the highest priority among the multiple relative positions is determined as the position information of the driving area, then according to the mapping between the position information in the pre-established image and the orientation of the digital person It is determined that the target orientation corresponding to the location information of the driving area is the southwest orientation, and the animation of the digital person displayed on the display device installed on the car turning to the southwest orientation is generated and displayed. Furthermore, it can also be combined with the attribute information of the object getting on and off the vehicle, which specifically refers to the attribute information of the object getting off the vehicle in the driving area, to generate and display the digital human broadcast.
  • the digital human can automatically broadcast the voice of "goodbye” or “take a break” , And/or make an expression or action of "byebye", or an animation of at least one of expression information, action information, and voice information corresponding to the attribute information of the object can also be displayed.
  • the relative position with the highest priority among the multiple relative positions includes the position information of the co-pilot area, then according to the position information in the pre-established image and the orientation of the digital person It is determined that the target orientation corresponding to the position information of the co-pilot area is the southeast orientation, and the animation of the digital person displayed in the display device installed on the car turning to the southeast orientation is generated and displayed.
  • the digital human can automatically broadcast the voice of "Welcome, so and so” , And/or make an expression or action of "applaud and welcome", or an animation of expression information, action information, and voice information corresponding to the attribute information of the object can also be displayed, such as an animation of "Welcome XX".
  • the relative position information can be determined, and then the target orientation corresponding to the relative position information can be determined, and an animation of the digital person turning to the target orientation can be generated and displayed.
  • an animation of the digital person broadcasting or displaying the multimedia information corresponding to the attribute information can be generated and displayed. For example, if the object is a pet, broadcast "Welcome, puppy" and so on.
  • the human-computer interaction method is more in line with human interaction habits, the interaction process is more natural, and people feel the warmth of human-computer interaction, and the ride pleasure, comfort, and sense of accompaniment are improved.
  • the present disclosure also provides an embodiment of the device.
  • FIG. 10 is a block diagram of a vehicle-mounted digital human interaction device according to an exemplary embodiment of the present disclosure.
  • the device includes: an acquisition module 710 for acquiring a video stream of an area in the vehicle; relative position detection The module 720 is configured to detect the relative position information of the object getting on and off the vehicle according to the image in the video stream.
  • the relative position information includes: the first relative position information of the object getting on and off the vehicle in the image or the up and down
  • the first interaction module 730 is configured to generate and display the animation of the digital person displayed in the display device provided on the car performing corresponding steering according to the relative position information.
  • the relative position detection module includes: a first determining sub-module, configured to determine, in an image coordinate system corresponding to the image, an image coordinate area where at least one part of the object getting on and off the vehicle is located;
  • the second determining sub-module is configured to use the image coordinate area as the first relative position information of the object getting on and off the vehicle in the image.
  • the relative position detection module includes: a third determining sub-module, configured to determine, in the image coordinate system corresponding to the image, the image coordinate area where at least one part of the object getting on and off the vehicle is located;
  • the fourth determining sub-module is used to determine the in-vehicle space coordinate area corresponding to the image coordinate area where at least one part of the object getting on and off the vehicle is located according to the mapping relationship between the image coordinates and the in-vehicle space coordinates;
  • the fifth determining sub-module The module is used to use the space coordinate area in the vehicle as the second relative position information of the object of getting on and off the vehicle and the vehicle.
  • the first interaction module includes: an orientation determination sub-module, configured to determine a target corresponding to the relative position information according to a pre-established mapping relationship between the position information in the image and the orientation of the digital person Orientation; a first interaction sub-module for generating and displaying the animation of the digital person turning to the target orientation displayed in the display device set on the vehicle.
  • the first interaction module includes: a sixth determining sub-module, configured to respond to the detected relative position information of objects getting on and off the vehicle, including information representing multiple relative positions of multiple objects getting on and off the vehicle , Determine the relative position of the target among the plurality of relative positions; the second interaction sub-module is used to generate and display the animation of the digital person displayed in the display device provided on the vehicle performing the corresponding steering according to the relative position of the target .
  • the sixth determining submodule includes: a determining unit, configured to respond to the detected relative position information of objects getting on and off the vehicle, including multiple relative positions of information representing multiple objects getting on and off the vehicle, according to The predetermined priority order of different position information is determined, and the relative position with the highest priority among the multiple relative positions is determined as the target relative position.
  • the priority order of the different location information includes at least one of the following: the priority of the location information corresponding to the driving area is higher than the priority of the location information corresponding to the co-pilot area; the location corresponding to the front row area The priority of the information is higher than the priority of the location information corresponding to the rear area; the priority of the location information corresponding to the dedicated seat is higher than the priority of the location information corresponding to the non-dedicated seat.
  • the device further includes: a boarding and boarding intention determination module, configured to determine the object’s boarding and boarding intention based on the multiple frames of images with time-series relationships in the video stream; and a boarding and boarding object determination module, configured to determine The getting on and off intention of the object determines the getting on and off object.
  • a boarding and boarding intention determination module configured to determine the object’s boarding and boarding intention based on the multiple frames of images with time-series relationships in the video stream
  • a boarding and boarding object determination module configured to determine The getting on and off intention of the object determines the getting on and off object.
  • the getting on and off intention determination module includes at least one of the following: a seventh determination sub-module, configured to respond to the detection of an object within a range that is less than a preset distance from the vehicle door in the multi-frame image At least one part determines the object’s intention to get off the car; the eighth determining sub-module is used to respond to the fact that no part of the object is detected within a range of less than a preset distance from the car door in the multi-frame images and the car door is in Open state, determine the subject's intention to get on the vehicle.
  • the device further includes: an attribute information detection module, configured to detect the attribute information of the object getting on and off the vehicle based on the image in the video stream;
  • the attribute information of the object of the vehicle generates and displays the animation of the digital person broadcasting or displaying the multimedia information corresponding to the attribute information.
  • the object to get on and off the vehicle includes at least one of the following: people and pets;
  • the attribute information of the object includes at least one of the following: identity information, gender information, and category information;
  • the multimedia information includes at least the following One: facial expression information, action information, and voice information.
  • the relevant part can refer to the part of the description of the method embodiment.
  • the device embodiments described above are merely illustrative, where the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place. , Or it can be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the objectives of the solutions of the present disclosure. Those of ordinary skill in the art can understand and implement without creative work.
  • the embodiment of the present disclosure also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the processor executes the computer program, the processor is used to execute the computer program described in any of the above-mentioned embodiments. Based on the interactive method of vehicle-mounted digital human.
  • the embodiments of the present disclosure provide a computer program product, including computer-readable code.
  • the processor in the device executes to achieve the same as that provided in any of the above embodiments.
  • the instruction of the interactive method based on the vehicle-mounted digital human.
  • the computer program product can be specifically implemented by hardware, software, or a combination thereof.
  • the computer program product is specifically embodied as a computer storage medium.
  • the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and so on.
  • SDK software development kit
  • the embodiment of the present disclosure also provides an interactive device based on a vehicle-mounted digital human, including: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to call the executable instructions stored in the memory At the time, the interaction method based on the vehicle-mounted digital human as described in any of the foregoing embodiments is implemented.
  • FIG. 11 is a schematic diagram of the hardware structure of an interactive device based on a vehicle-mounted digital human provided by an embodiment of the application.
  • the vehicle-mounted digital human-based interaction device 810 includes a processor 811, and may also include an input device 812, an output device 813, and a memory 814.
  • the input device 812, output device 813, memory 814, and processor 811 are connected to each other through a bus.
  • the memory 814 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or portable Read-only memory (compact disc read-only memory, CD-ROM), the memory 814 is used to store related instructions and data.
  • RAM random access memory
  • ROM read-only memory
  • EPROM erasable programmable read-only memory
  • CD-ROM compact disc read-only memory
  • the input device 812 is used to input data and/or signals
  • the output device 813 is used to output data and/or signals.
  • the output device 813 and the input device 812 may be independent devices or a whole device.
  • the output device 813 includes a display device 8131 and other output devices 8132.
  • the other output devices 8132 may include, but are not limited to, audio output devices.
  • the digital person is displayed through the display device 8131, and the digital person performs the corresponding steering action.
  • the animation of is also displayed on the display device 8131.
  • the processor 811 may include one or more processors, for example, one or more central processing units (CPU).
  • CPU central processing units
  • the processor 811 is a CPU
  • the CPU may be a single-core CPU or It is a multi-core CPU.
  • the memory 814 is used to store program codes and data of the network device.
  • the processor 811 is configured to call the program code and data in the memory 814 to execute the steps in the foregoing method embodiment. For details, please refer to the description in the method embodiment, which will not be repeated here.
  • FIG. 11 only shows a simplified design of an interactive device based on a vehicle-mounted digital human.
  • the vehicle-mounted digital human-based interaction device can also contain other necessary components, including but not limited to any number of input/output devices, processors, controllers, memories, etc., and all of them can implement the embodiments of this application. All of the interactive devices based on vehicle-mounted digital humans are within the protection scope of this application.
  • the functions or modules contained in the device provided in the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments.
  • the functions or modules contained in the device provided in the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)
  • Navigation (AREA)

Abstract

一种基于车载数字人的交互方法及装置、存储介质，其中，该方法包括：获取车内区域的视频流（101）；根据视频流中的图像，检测上下车的对象的相对位置信息（102），所述相对位置信息包括：所述上下车的对象在所述图像中的第一相对位置信息或者所述上下车的对象与车的第二相对位置信息；根据所述相对位置信息，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画（103）。

Description

基于车载数字人的交互方法及装置、存储介质
相关申请的交叉引用
本公开要求在2020年6月23日提交中国专利局、申请号为CN 202010581363.4、发明名称为“基于车载数字人的交互方法及装置、存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本公开涉及增强现实领域，尤其涉及一种基于车载数字人的交互方法及装置、存储介质。
背景技术
目前，可以在车内放置机器人，在人员进入车内后，通过机器人与车内人员进行交互。但是机器人与车内人员的交互模式比较固定，缺少人性化。
发明内容
本公开提供了一种基于车载数字人的交互方法及装置、存储介质。
根据本公开实施例的第一方面，提供一种基于车载数字人的交互方法，所述方法包括：获取车内区域的视频流；根据视频流中的图像，检测上下车的对象的相对位置信息，所述相对位置信息包括：所述上下车的对象在所述图像中的第一相对位置信息或者所述上下车的对象与车的第二相对位置信息；根据所述相对位置信息，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画。
在一些实施例中，所述根据视频流中的图像，检测上下车的对象的相对位置信息，包括：在所述图像对应的图像坐标系中，确定所述上下车的对象的至少一个部位所在的图像坐标区域；将所述图像坐标区域作为所述上下车的对象在所述图像中的所述第一相对位置信息。
在一些实施例中，所述根据视频流中的图像，检测上下车的对象的相对位置信息，包括：在所述图像对应的图像坐标系中，确定所述上下车的对象至少一个部位所在的图像坐标区域；根据图像坐标和车内空间坐标之间的映射关系，确定所述上下车的对象至 少一个部位所在的所述图像坐标区域对应的车内空间坐标区域；将所述车内空间坐标区域作为所述上下车的对象与车的所述第二相对位置信息。
在一些实施例中，所述根据所述相对位置信息，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画，包括：根据预先建立的图像中位置信息和数字人的朝向之间的映射关系，确定与所述相对位置信息对应的目标朝向；生成并显示所述车上设置的显示设备中显示的数字人转向所述目标朝向的动画。
在一些实施例中，所述根据所述相对位置信息，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画包括：响应于检测到的上下车的对象的相对位置信息包括多个上下车的对象的多个相对位置，确定所述多个相对位置中的目标相对位置；根据所述目标相对位置，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画。
在一些实施例中，所述响应于检测到的上下车的对象的相对位置信息包括多个上下车的对象的多个相对位置，确定所述多个相对位置中的目标相对位置，包括：响应于检测到的上下车的对象的相对位置信息包括多个上下车的对象的多个相对位置，根据预先确定的不同位置信息的优先级顺序，将所述多个相对位置中优先级最高的相对位置确定为所述目标相对位置。
在一些实施例中，所述不同位置信息的优先级顺序包括以下至少之一：对应驾驶区域的位置信息的优先级，高于对应副驾驶区域的位置信息的优先级；对应前排区域的位置信息的优先级，高于对应后排区域的位置信息的优先级；对应专用座椅的位置信息的优先级，高于对应非专用座椅的位置信息的优先级。
在一些实施例中，所述根据视频流中的图像，检测上下车的对象的相对位置信息之前，所述方法还包括：基于所述视频流中具备时序关系的多帧图像，确定对象的上下车意图；根据所述对象的上下车意图，确定所述上下车的对象。
在一些实施例中，所述基于所述视频流中具备时序关系的多帧图像，确定对象的上下车意图，包括以下至少之一：响应于在所述多帧图像中与车门距离小于预设距离的范围内检测到对象的至少一个部位，确定对象的下车意图；响应于在所述多帧图像中与车门距离小于预设距离的范围内未检测到对象的任何部位且所述车门处于开启状态，确定对象的上车意图。
在一些实施例中，所述方法还包括：基于所述视频流中的图像，检测所述上下车的 对象的属性信息；根据所述上下车的对象的属性信息，生成并显示所述数字人播报或展示与所述属性信息对应的多媒体信息的动画。
在一些实施例中，所述上下车的对象包括以下至少之一：人、宠物；所述对象的属性信息包括以下至少之一：身份信息、性别信息、类别信息；所述多媒体信息包括以下至少之一：表情信息、动作信息、语音信息。
根据本公开实施例的第二方面，提供一种基于车载数字人的交互装置，所述装置包括：获取模块，用于获取车内区域的视频流；相对位置检测模块，用于根据视频流中的图像，检测上下车的对象的相对位置信息，所述相对位置信息包括：所述上下车的对象在所述图像中的第一相对位置信息或者所述上下车的对象与车的第二相对位置信息；第一交互模块，用于根据所述相对位置信息，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画。
在一些实施例中，所述相对位置检测模块包括：第一确定子模块，用于在所述图像对应的图像坐标系中，确定所述上下车的对象的至少一个部位所在的图像坐标区域；第二确定子模块，用于将所述图像坐标区域作为所述上下车的对象在所述图像中的所述第一相对位置信息。
在一些实施例中，所述相对位置检测模块包括：第三确定子模块，用于在所述图像对应的图像坐标系中，确定所述上下车的对象至少一个部位所在的图像坐标区域；第四确定子模块，用于根据图像坐标和车内空间坐标之间的映射关系，确定所述上下车的对象至少一个部位所在的所述图像坐标区域对应的车内空间坐标区域；第五确定子模块，用于将所述车内空间坐标区域作为所述上下车的对象与车的所述第二相对位置信息。
在一些实施例中，所述第一交互模块包括：朝向确定子模块，用于根据预先建立的图像中位置信息和数字人的朝向之间的映射关系，确定与所述相对位置信息对应的目标朝向；第一交互子模块，用于生成并显示所述车上设置的显示设备中显示的数字人转向所述目标朝向的动画。
在一些实施例中，所述第一交互模块包括：第六确定子模块，用于响应于检测到的上下车的对象的相对位置信息包括多个上下车的对象的多个相对位置，确定所述多个相对位置中的目标相对位置；第二交互子模块，用于根据所述目标相对位置，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画。
在一些实施例中，所述第六确定子模块包括：确定单元，用于响应于检测到的上下 车的对象的相对位置信息包括多个上下车的对象的多个相对位置，根据预先确定的不同位置信息的优先级顺序，将所述多个相对位置中优先级最高的相对位置确定为所述目标相对位置。
在一些实施例中，所述不同位置信息的优先级顺序包括以下至少之一：对应驾驶区域的位置信息的优先级，高于对应副驾驶区域的位置信息的优先级；对应前排区域的位置信息的优先级，高于对应后排区域的位置信息的优先级；对应专用座椅的位置信息的优先级，高于对应非专用座椅的位置信息的优先级。
在一些实施例中，所述装置还包括：上下车意图确定模块，用于基于所述视频流中具备时序关系的多帧图像，确定对象的上下车意图；上下车对象确定模块，用于根据所述对象的上下车意图，确定所述上下车的对象。
在一些实施例中，所述上下车意图确定模块包括以下至少之一：第七确定子模块，用于响应于在所述多帧图像中与车门距离小于预设距离的范围内检测到对象的至少一个部位，确定对象的下车意图；第八确定子模块，用于响应于在所述多帧图像中与车门距离小于预设距离的范围内未检测到对象的任何部位且所述车门处于开启状态，确定对象的上车意图。
在一些实施例中，所述装置还包括：属性信息检测模块，用于基于所述视频流中的图像，检测所述上下车的对象的属性信息；第二交互模块，用于根据所述上下车的对象的属性信息，生成并显示所述数字人播报或展示与所述属性信息对应的多媒体信息的动画。
在一些实施例中，所述上下车的对象包括以下至少之一：人、宠物；所述对象的属性信息包括以下至少之一：身份信息、性别信息、类别信息；所述多媒体信息包括以下至少之一：表情信息、动作信息、语音信息。
根据本公开实施例的第三方面，提供一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，当处理器执行所述计算机程序时，所述处理器用于执行上述第一方面任一实施例所述的基于车载数字人的交互方法。
根据本公开实施例的第四方面，提供一种基于车载数字人的交互装置，包括：处理器；用于存储所述处理器可执行指令的存储器；其中，所述处理器被配置为调用所述存储器中存储的可执行指令时，实现第一方面中任一实施例所述的基于车载数字人的交互方法。
根据本公开实施例的第五方面，提供一种计算机程序产品，包括计算机可读代码，当计算机可读代码在设备上运行时，设备中的处理器执行用于实现如上任一实施例提供的基于车载数字人的交互方法的指令。
本公开的实施例提供的技术方案可以包括以下有益效果：
本公开实施例中，通过对车内区域的视频流中的图像进行分析，检测上下车的对象的相对位置信息，并根据相对位置信息，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画。通过对视频流中的图像进行检测分析的方式，在有对象上下车的情况下，生成并显示数字人执行相应转向的动画，使得人机交互的方式更加符合人的交互习惯，交互过程更加自然，让人感受到人机交互的温暖，提升乘车乐趣、舒适感和陪护感。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本公开。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。
图1是本公开根据一示例性实施例示出的一种基于车载数字人的交互方法流程图；
图2是本公开根据一示例性实施例示出的步骤102的流程图；
图3是本公开根据另一示例性实施例示出的步骤102的流程图；
图4是本公开根据一示例性实施例示出的步骤103的流程图；
图5是本公开根据另一示例性实施例示出的步骤103的流程图；
图6是本公开根据另一示例性实施例示出的一种基于车载数字人的交互方法流程图；
图7是本公开根据一示例性实施例示出的步骤104的流程图；
图8是本公开根据再一示例性实施例示出的一种基于车载数字人的交互方法流程图；
图9是本公开根据一示例性实施例示出的一种车载数字人的示意图；
图10是本公开根据一示例性实施例示出的一种基于车载数字人的交互装置框图；
图11是本公开根据一示例性实施例示出的一种基于车载数字人的交互装置的硬件 结构示意图。
具体实施方式
这里将详细地对示例性实施例进行说明，其示例表示在附图中。下面的描述涉及附图时，除非另有表示，不同附图中的相同数字表示相同或相似的要素。以下示例性实施例中所描述的实施方式并不代表与本公开相一致的所有实施方式。相反，它们仅是与如所附权利要求书中所详述的、本公开的一些方面相一致的装置和方法的例子。
在本公开运行的术语是仅仅出于描述特定实施例的目的，而非旨在限制本公开。在本公开和所附权利要求书中所运行的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义。还应当理解，本文中运行的术语“和/或”是指并包含一个或多个相关联的列出项目的任何或所有可能组合。
应当理解，尽管在本公开可能采用术语第一、第二、第三等来描述各种信息，但这些信息不应限于这些术语。这些术语仅用来将同一类型的信息彼此区分开。例如，在不脱离本公开范围的情况下，第一信息也可以被称为第二信息，类似地，第二信息也可以被称为第一信息。取决于语境，如在此所运行的词语“如果”可以被解释成为“在……时”或“当……时”或“响应于确定”。
本公开实施例提供了一种基于车载数字人的交互方法，可以用于可驾驶的机器设备，例如智能车辆、模拟车辆驾驶的智能车舱等。
如图1所示，图1是根据一示例性实施例示出的一种基于车载数字人的交互方法，包括以下步骤：
在步骤101中，获取车内区域的视频流。
在本公开实施例中，可以通过车载摄像头获取车内区域的视频流。其中，车载摄像头可以设置在中控台上、前风挡玻璃上、车内后视镜上、或者其他可以拍摄到车内区域的任意位置。车内区域包括以下至少之一：驾驶区域、副驾驶区域、后排区域。
在步骤102中，根据视频流中的图像，检测上下车的对象的相对位置信息。
在本公开实施例中，上下车的对象包括以下至少之一：人、宠物。相对位置信息包括第一相对位置信息或者第二相对位置信息，其中，第一相对位置信息是上下车的对象在视频流的图像中的相对位置信息，可以用对象在图像坐标系中的图像坐标区域来表示，第二相对位置信息是上下车的对象与车的相对位置信息，可以用上下车的对象在车内空 间坐标系中的车内空间坐标区域表示。
在步骤103中，根据所述相对位置信息，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画。
在本公开实施例中，所述数字人是通过软件生成的虚拟形象，可在车上设置的显示设备上显示，其中显示设备可以包括但不限于中控显示屏或车载平板设备。在确定了上下车的对象的相对位置信息后，可以根据该相对位置信息，生成并显示显示设备中显示的数字人执行相应转向的动画，例如，生成并显示数字人从之前的朝向转向上下车的对象所在朝向的动画。
上述实施例中，通过对车内区域的视频流中的图像进行分析，检测上下车的对象的相对位置信息，并根据相对位置信息，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画。通过对视频流中的图像进行检测分析的方式，在有对象上下车的情况下，生成并显示数字人执行相应转向的动画，使得人机交互的方式更加符合人的交互习惯，交互过程更加自然，让人感受到人机交互的温暖，提升乘车乐趣、舒适感和陪护感。例如，在有人上车的情况下，通过本公开实施例提供的基于车载数字人的交互方法，无需该人进行任何额外的语言或者动作的输入，即可准确的判断来人的方向，实现真正的无感识别和交互。
在一些实施例中，在上下车的对象的相对位置信息包括该上下车的对象在图像中的第一相对位置信息的情况下，例如图2所示，步骤102可以包括：
在步骤201中，在所述图像对应的图像坐标系中，确定所述上下车的对象的至少一个部位所在的图像坐标区域。
在本公开实施例中，图像坐标系是视频流中的图像所对应的二维坐标系。在一个示例中，可以通过预先训练好的区域生成网络(Region Proposal Network，RPN)对图像中上下车的对象的至少一个部位进行区域检测，至少一个部位可以包括但不限于脸部、手部、躯干等。通过区域检测，可以确定出上下车的对象的至少一个部位所在的图像坐标区域。
在步骤202中，将所述图像坐标区域作为所述上下车的对象在所述图像中的所述第一相对位置信息。
例如，上下车的对象至少一个部位所在的所述图像坐标区域为图像坐标系中的图像坐标区域1，则将图像坐标区域1作为上下车的对象在所述图像中的所述第一相对位置 信息。
上述实施例中，可以通过对上下车的对象进行区域检测，从而确定出上下车的对象的至少一个部位所在的图像坐标区域，并将确定出的图像坐标区域作为第一相对位置，以便后续根据第一相对位置，生成并显示数字人执行相应转向的动画，可用性高。
在一些实施例中，在上下车的对象的相对位置信息包括该上下车的对象与车的第二相对位置信息的情况下，例如图3所示，步骤102可以包括：
在步骤301中，在所述图像对应的图像坐标系中，确定所述上下车的对象至少一个部位所在的图像坐标区域。
在本公开实施例中，可以采用与步骤201相同的方式确定上下车的对象至少一个部位所在的图像坐标区域，在此不再赘述。
在步骤302中，根据图像坐标和车内空间坐标之间的映射关系，确定所述上下车的对象至少一个部位所在的所述图像坐标区域对应的车内空间坐标区域。
在本公开实施例中，车内空间坐标区域是在车内空间坐标系中的坐标区域，车内空间坐标系是覆盖了车内区域的三维空间坐标系。
在本公开实施例中，预先确定了图像坐标和车内空间坐标之间的映射关系，在确定了上下车的对象至少一个部位所在的所述图像坐标区域后，可以根据上述映射关系，确定上下车的对象对应的车内空间坐标区域。
在步骤303中，将所述车内空间坐标区域作为所述上下车的对象与车的所述第二相对位置信息。
例如，上下车的对象至少一个部位所在的所述图像坐标区域为图像坐标系中的图像坐标区域1，根据映射关系，图像坐标区域1对应的车内空间坐标区域为驾驶区域，则将驾驶区域作为上下车的对象与车的第二相对位置信息。
上述实施例中，可以在确定出上下车的对象的至少一个部位所在的图像坐标区域的基础上，根据图像坐标和车内空间坐标之间的映射关系，确定上下车的对象对应的车内空间坐标区域，将该车内空间坐标区域作为上下车的对象与车的第二相对位置，以便后续根据第二相对位置，生成并显示数字人执行相应转向的动画，可用性高。
在一些实施例中，例如图4所示，步骤103可以包括：
在步骤401中，根据预先建立的图像中位置信息和数字人的朝向之间的映射关系， 确定与所述相对位置信息对应的目标朝向。
在本公开实施例中，预先建立了图像中位置信息和数字人的朝向之间的映射关系，例如，图像中的属于区域位置1的位置信息对应的数字人的朝向为西南朝向，属于区域位置2的位置信息对应的数字人的朝向为东南朝向等，根据这个映射关系，在确定了相对位置信息后，可以确定出对应的目标朝向。
在步骤402中，生成并显示所述车上设置的显示设备中显示的数字人转向所述目标朝向的动画。
在本公开实施例中，显示设备上已经显示了数字人，根据确定的目标朝向，可以生成让数字人转向目标朝向的动画，并显示该动画。例如，数字人之前是朝向正南方向的，目标朝向为西南方向，生成并显示的动画为数字人由正南方向转向西南方向的动画。
上述实施例中，根据图像中位置信息和数字人的朝向之间的映射关系，可以确定与上下车对象的相对位置信息对应的目标朝向，生成并显示数字人转向所述目标朝向的动画。从而在有对象上下车的情况下，让数字人转向对应的目标朝向，使得人机交互的方式更加符合人的交互习惯，交互过程更加自然，让人感受到人机交互的温暖，提升乘车乐趣、舒适感和陪护感。
在一些实施例中，例如图5所示，步骤103可以包括：
在步骤501中，响应于检测到的上下车的对象的相对位置信息包括表征多个上下车的对象的多个相对位置的信息，确定所述多个相对位置中的目标相对位置。
在本公开实施例中，在检测到的上下车的对象的相对位置信息包括表征多个相对位置的信息的情况下，例如同时有多个对象上车，或者下车，或者有的对象上车、有的对象下车，可以确定多个相对位置中的目标相对位置。
在步骤502中，根据所述目标相对位置，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画。
在本公开实施例中，根据多个相对位置中的目标相对位置，来生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画。
例如，同时有多个对象上车，则生成并显示数字人执行转向目标朝向的动画，该目标朝向是目标相对位置所对应的朝向。
在一些实施例中，步骤501包括：
响应于检测到的上下车的对象的相对位置信息包括表征多个上下车的对象的多个相对位置的信息，根据预先确定的不同位置信息的优先级顺序，将所述多个相对位置中优先级最高的相对位置确定为目标相对位置。
在本公开实施例中，不同位置信息的优先级顺序包括以下至少之一：对应驾驶区域的位置信息的优先级，高于对应副驾驶区域的位置信息的优先级；对应前排区域的位置信息的优先级，高于对应后排区域的位置信息的优先级；对应专用座椅的位置信息的优先级，高于对应非专用座椅的位置信息的优先级。
其中，根据上述优先级顺序，如果表征多个上下车的对象的多个相对位置的信息中至少包括驾驶区域的位置信息，则驾驶区域的位置信息的优先级高于其他区域的位置信息的优先级。如果表征多个上下车的对象的多个相对位置的信息中不包括驾驶区域的位置信息，但是包括副驾驶区域和后排区域的位置信息，那么副驾驶区域的位置信息的优先级高于后排区域的位置信息的优先级。
专用座椅可以包括但不限于儿童专用座椅、其他人士专用座椅等，如果车内安装了专用座椅，可以认为专用座椅的位置信息的优先级，高于对应非专用座椅的位置信息的优先级。
根据上述优先级顺序，可以将多个相对位置中优先级最高的相对位置确定为目标相对位置，后续根据这个目标相对位置，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画。
上述实施例中，在检测到的上下车的对象的相对位置信息包括表征多个上下车的对象的多个相对位置的信息的情况下，可以根据预先确定的不同位置信息的优先级顺序，将所述多个相对位置中优先级最高的相对位置确定为目标相对位置，从而根据目标相对位置，确定数字人对应的目标朝向，生成并显示数字人转向目标朝向的动画，从而在上下车的对象为多个的情况下，让数字人转向优先级高的对象，使得人机交互的方式更加符合人的交互习惯，交互过程更加自然，让人感受到人机交互的温暖，提升乘车乐趣、舒适感和陪护感。
在一些实施例中，例如图6所示，在执行步骤102之前，该方法还可以包括：
在步骤104中，基于所述视频流中具备时序关系的多帧图像，确定对象的上下车意图。
在本公开实施例中，具备时序关系的多帧图像可以是在时序上相邻的多帧图像。
在步骤105中，根据所述对象的上下车意图，确定所述上下车的对象。
在本公开实施例中，根据对象的上车意图，确定该对象是上车的对象，根据对象的下车意图，可以确定该对象是下车的对象。
在一些实施例中，例如图7所示，步骤104可以包括以下至少之一：
在步骤601中，响应于在所述多帧图像中与车门距离小于预设距离的范围内检测到对象的至少一个部位，确定对象的下车意图。
在本公开实施例中，如果在多帧图像中，相距任意一个车门的距离小于预设距离的范围内，检测到有对象的至少一个部位，例如手部、头部、腿部等，可以确定对象的下车意图。例如，先检测任意一个车门所对应的像素区域，在距离这个像素区域预设距离的范围内，检测到有某个对象的手部或者头部或者腿部，则可以确定该对象的下车意图。
在步骤602中，响应于在所述多帧图像中与车门距离小于预设距离的范围内未检测到对象的任何部位且所述车门处于开启状态，确定对象的上车意图。
在本公开实施例中，如果在多帧图像中，先检测任意一个车门所对应的像素区域，在距离这个像素区域预设距离的范围内，未检测到对象的任何部位，说明目前未有对象通过该车门上车，同时根据多帧图像确定车门处于开启状态，那么可以认为有对象希望通过打开该车门进入车内，此时可以确定对象的上车意图。
在上述实施例中，为了避免误判，还可以结合车上的传感器的检测结果进一步确定对象的上下车意图。传感器包括但不限于车门传感器、座椅上的压力传感器等。
在一个示例中，车门传感器显示车门处于解锁状态，座椅上安装的压力传感器显示有对象正处于座椅上，此时如果在多帧图像中与车门距离小于预设距离的范围内检测到对象的至少一个部位，那么可以确定该对象的下车意图。
在另一个示例中，车门传感器显示车门处于解锁状态，座椅上安装的压力传感器显示没有对象处于座椅上，此时如果在所述多帧图像中与车门距离小于预设距离的范围内未检测到对象的任何部位且所述车门处于开启状态，那么可以确定对象的上车意图。
上述实施例中，可以结合视频流中具备时序关系的多帧图像，确定对象的上下车意图，从而确定上下车的对象，以便后续生成并显示数字人执行转向的动画，可用性高。
在一些实施例中，例如图8所示，上述方法还可以包括：
在步骤106中，基于所述视频流中的图像，检测所述上下车的对象的属性信息。
在本公开实施例中，通过对视频流中的图像进行人脸识别，提取出上下车的对象的人脸特征，通过将提取出的人脸特征与预存的人脸特征进行比对，确定比对结果，从而可以确定上下车的对象的属性信息，对象的属性信息包括以下至少之一：身份信息、性别信息、类别信息。其中，身份信息用于指示上下车的对象的具体身份，例如张三、李四，性别信息用于指示上下车的对象的性别，例如男、女，类型信息用于指示上下车的对象的类型，例如类型属于大人、小孩还是宠物。
在步骤107中，根据所述上下车的对象的属性信息，生成并显示所述数字人播报或展示与所述属性信息对应的多媒体信息的动画。
在本公开实施例中，多媒体信息包括以下至少之一：表情信息、动作信息、语音信息。
在一个示例中，上下车的对象的属性信息包括身份信息为张三，性别信息为女，类型信息为大人，则可以在显示设备上生成并显示与该身份信息对应的数字人，例如图9所示，进一步地，生成并显示所述数字人播报或展示与所述属性信息对应的表情、动作、语音至少之一的动画。
例如，数字人可以与车内人员聊天，通过包括语音、表情、动作至少之一的动画向车内人员问好，“上午好，张三”。
如果身份信息指示该对象首次上车，可以结合该人员的性别信息和类别信息中的至少一项，进行播报或通过展示动画来问好，例如“早上好，女士”。
在另一个示例中，数字人可以自动通过播报动画来询问是否需要介绍车况等信息，如果车内人员回答“是”，数字人可以介绍车况，如果车内人员回答或“否”，则数字人可以进入待机状态，数字人可以通过语音、动作、表情至少之一的动画，表示自己进入待机状态。
上述实施例中，通过检测得到的上下车的对象的属性信息，生成并显示数字人播报或展示与属性信息对应的多媒体信息的动画，使得数字人与车内人员的交互更加多模式化，提升了数字人的智能化程度。
在一个示例中，同时有多个对象下车，确定多个相对位置中优先级最高的相对位置为驾驶区域的位置信息，那么根据预先建立的图像中位置信息和数字人的朝向之间的映射关系，确定驾驶区域的位置信息对应的目标朝向为西南朝向，生成并显示车上设 置的显示设备中显示的数字人转向西南朝向的动画。进一步地，还可以结合上下车的对象的属性信息，这里具体指驾驶区域下车的对象的属性信息，生成并显示数字人播报，例如数字人可以自动播报“再见”或者“休息一下”的语音，和/或作出“byebye”的表情或动作，或者还可以展示与该对象的属性信息对应的表情信息、动作信息、语音信息至少之一的动画。
在另一个示例中，同时有多个对象上车，确定多个相对位置中优先级最高的相对位置包括副驾驶区域的位置信息，那么根据预先建立的图像中位置信息和数字人的朝向之间的映射关系，确定副驾驶区域的位置信息对应的目标朝向为东南朝向，生成并显示车上设置的显示设备中显示的数字人转向东南朝向的动画。进一步地，还可以结合上车的对象的属性信息，这里可以指副驾驶区域上车的对象的属性信息，生成并显示数字人播报，例如数字人可以自动播报“欢迎你，某某”的语音，和/或作出“鼓掌欢迎”的表情或动作，或者还可以展示与该对象的属性信息对应的表情信息、动作信息、语音信息的动画，例如“欢迎某某”的动画。
在另一个示例中，如果只有一个对象上车或下车，那么可以确定其相对位置信息，进而确定与该相对位置信息对应的目标朝向，生成并显示数字人转向该目标朝向的动画。当然，在该对象上车或下车的情况下，可以生成并显示数字人播报或展示与所述属性信息对应的多媒体信息的动画。例如，对象是宠物，播报“欢迎你，puppy”等。
上述实施例中，人机交互的方式更加符合人的交互习惯，交互过程更加自然，让人感受到人机交互的温暖，提升乘车乐趣、舒适感和陪护感。
与前述方法实施例相对应，本公开还提供了装置的实施例。
如图10所示，图10是本公开根据一示例性实施例示出的一种基于车载数字人的交互装置框图，装置包括：获取模块710，用于获取车内区域的视频流；相对位置检测模块720，用于根据视频流中的图像，检测上下车的对象的相对位置信息，所述相对位置信息包括：所述上下车的对象在所述图像中的第一相对位置信息或者所述上下车的对象与车的第二相对位置信息；第一交互模块730，用于根据所述相对位置信息，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画。
在一些实施例中，所述相对位置检测模块包括：第一确定子模块，用于在所述图像对应的图像坐标系中，确定所述上下车的对象的至少一个部位所在的图像坐标区域；第二确定子模块，用于将所述图像坐标区域作为所述上下车的对象在所述图像中的所述 第一相对位置信息。
在一些实施例中，所述相对位置检测模块包括：第三确定子模块，用于在所述图像对应的图像坐标系中，确定所述上下车的对象至少一个部位所在的图像坐标区域；第四确定子模块，用于根据图像坐标和车内空间坐标之间的映射关系，确定所述上下车的对象至少一个部位所在的所述图像坐标区域对应的车内空间坐标区域；第五确定子模块，用于将所述车内空间坐标区域作为所述上下车的对象与车的所述第二相对位置信息。
在一些实施例中，所述第一交互模块包括：朝向确定子模块，用于根据预先建立的图像中位置信息和数字人的朝向之间的映射关系，确定与所述相对位置信息对应的目标朝向；第一交互子模块，用于生成并显示所述车上设置的显示设备中显示的数字人转向所述目标朝向的动画。
在一些实施例中，所述第一交互模块包括：第六确定子模块，用于响应于检测到的上下车的对象的相对位置信息包括表征多个上下车的对象的多个相对位置的信息，确定所述多个相对位置中的目标相对位置；第二交互子模块，用于根据所述目标相对位置，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画。
在一些实施例中，所述第六确定子模块包括：确定单元，用于响应于检测到的上下车的对象的相对位置信息包括表征多个上下车的对象的多个相对位置的信息，根据预先确定的不同位置信息的优先级顺序，将所述多个相对位置中优先级最高的相对位置确定为所述目标相对位置。
在一些实施例中，所述不同位置信息的优先级顺序包括以下至少之一：对应驾驶区域的位置信息的优先级，高于对应副驾驶区域的位置信息的优先级；对应前排区域的位置信息的优先级，高于对应后排区域的位置信息的优先级；对应专用座椅的位置信息的优先级，高于对应非专用座椅的位置信息的优先级。
在一些实施例中，所述装置还包括：上下车意图确定模块，用于基于所述视频流中具备时序关系的多帧图像，确定对象的上下车意图；上下车对象确定模块，用于根据所述对象的上下车意图，确定所述上下车的对象。
在一些实施例中，所述上下车意图确定模块包括以下至少之一：第七确定子模块，用于响应于在所述多帧图像中与车门距离小于预设距离的范围内检测到对象的至少一个部位，确定对象的下车意图；第八确定子模块，用于响应于在所述多帧图像中与车门距离小于预设距离的范围内未检测到对象的任何部位且所述车门处于开启状态，确定 对象的上车意图。
在一些实施例中，所述装置还包括：属性信息检测模块，用于基于所述视频流中的图像，检测所述上下车的对象的属性信息；第二交互模块，用于根据所述上下车的对象的属性信息，生成并显示所述数字人播报或展示与所述属性信息对应的多媒体信息的动画。
在一些实施例中，所述上下车的对象包括以下至少之一：人、宠物；所述对象的属性信息包括以下至少之一：身份信息、性别信息、类别信息；所述多媒体信息包括以下至少之一：表情信息、动作信息、语音信息。
对于装置实施例而言，由于其基本对应于方法实施例，所以相关之处参见方法实施例的部分说明即可。以上所描述的装置实施例仅仅是示意性的，其中作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本公开方案的目的。本领域普通技术人员在不付出创造性劳动的情况下，即可以理解并实施。
本公开实施例还提供了一种计算机可读存储介质，该计算机可读存储介质存储有计算机程序，当处理器执行所述计算机程序时，所述处理器用于执行上述任一实施例所述的基于车载数字人的交互方法。
在一些实施例中，本公开实施例提供了一种计算机程序产品，包括计算机可读代码，当计算机可读代码在设备上运行时，设备中的处理器执行用于实现如上任一实施例提供的基于车载数字人的交互方法的指令。
该计算机程序产品可以具体通过硬件、软件或其结合的方式实现。在一个实施例中，所述计算机程序产品具体体现为计算机存储介质，在另一个实施例中，计算机程序产品具体体现为软件产品，例如软件开发包(Software Development Kit，SDK)等等。
本公开实施例还提供了一种基于车载数字人的交互装置，包括：处理器；用于存储处理器可执行指令的存储器；其中，处理器被配置为调用所述存储器中存储的可执行指令时，实现上述任一实施例所述的基于车载数字人的交互方法。
图11为本申请实施例提供的一种基于车载数字人的交互装置的硬件结构示意图。该基于车载数字人的交互装置810包括处理器811，还可以包括输入装置812、输出装置813和存储器814。该输入装置812、输出装置813、存储器814和处理器811之间通 过总线相互连接。
存储器814包括但不限于随机存储记忆体(random access memory，RAM)、只读存储器(read-only memory，ROM)、可擦除可编程只读存储器(erasable programmable read only memory，EPROM)、或便携式只读存储器(compact disc read-only memory，CD-ROM)，该存储器814用于存储相关指令及数据。
输入装置812用于输入数据和/或信号，以及输出装置813用于输出数据和/或信号。输出装置813和输入装置812可以是独立的器件，也可以是一个整体的器件。
在本公开实施例中，输出装置813包括显示设备8131和其他输出设备8132，其他输出设备8132可以包括但不限于音频输出设备，数字人是通过显示设备8131进行显示的，数字人执行相应转向动作的动画同样是通过显示设备8131来显示。
处理器811可以包括一个或多个处理器，例如包括一个或多个中央处理器(central processing unit，CPU)，在处理器811是一个CPU的情况下，该CPU可以是单核CPU，也可以是多核CPU。
存储器814用于存储网络设备的程序代码和数据。
处理器811用于调用该存储器814中的程序代码和数据，执行上述方法实施例中的步骤。具体可参见方法实施例中的描述，在此不再赘述。
可以理解的是，图11仅仅示出了一种基于车载数字人的交互装置的简化设计。在实际应用中，基于车载数字人的交互装置还可以分别包含必要的其他元件，包含但不限于任意数量的输入/输出装置、处理器、控制器、存储器等，而所有可以实现本申请实施例的基于车载数字人的交互装置都在本申请的保护范围之内。
在一些实施例中，本公开实施例提供的装置具有的功能或包含的模块可以用于执行上文方法实施例描述的方法，其具体实现可以参照上文方法实施例的描述，为了简洁，这里不再赘述。
本领域技术人员在考虑说明书及实践这里公开的发明后，将容易想到本公开的其它实施方案。本公开旨在涵盖本公开的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本公开的一般性原理并包括本公开未公开的本技术领域中的公知常识或者惯用技术手段。说明书和实施例仅被视为示例性的，本公开的真正范围和精神由下面的权利要求指出。
以上所述仅为本公开的一些实施例而已，并不用以限制本公开，凡在本公开的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本公开保护的范围之内。

Claims (15)

  1. 一种基于车载数字人的交互方法，包括：
    获取车内区域的视频流；
    根据视频流中的图像，检测上下车的对象的相对位置信息，所述相对位置信息包括：所述上下车的对象在所述图像中的第一相对位置信息或者所述上下车的对象与车的第二相对位置信息；
    根据所述相对位置信息，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画。
  2. 根据权利要求1所述的方法，其中，所述根据视频流中的图像，检测上下车的对象的相对位置信息，包括：
    在所述图像对应的图像坐标系中，确定所述上下车的对象的至少一个部位所在的图像坐标区域；
    将所述图像坐标区域作为所述上下车的对象在所述图像中的所述第一相对位置信息。
  3. 根据权利要求1所述的方法，其中，所述根据视频流中的图像，检测上下车的对象的相对位置信息，包括：
    在所述图像对应的图像坐标系中，确定所述上下车的对象至少一个部位所在的图像坐标区域；
    根据图像坐标和车内空间坐标之间的映射关系，确定所述上下车的对象至少一个部位所在的所述图像坐标区域对应的车内空间坐标区域；
    将所述车内空间坐标区域作为所述上下车的对象与车的所述第二相对位置信息。
  4. 根据权利要求1-3任一项所述的方法，其中，所述根据所述相对位置信息，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画，包括：
    根据预先建立的图像中位置信息和数字人的朝向之间的映射关系，确定与所述相对位置信息对应的目标朝向；
    生成并显示所述车上设置的显示设备中显示的数字人转向所述目标朝向的动画。
  5. 根据权利要求1-4任一项所述的方法，其中，所述根据所述相对位置信息，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画，包括：
    响应于检测到的上下车的对象的相对位置信息包括表征多个上下车的对象的多个相对位置的信息，确定所述多个相对位置中的目标相对位置；
    根据所述目标相对位置，生成并显示所述车上设置的显示设备中显示的数字人执行 相应转向的动画。
  6. 根据权利要求5所述的方法，其中，所述响应于检测到的上下车的对象的相对位置信息包括表征多个上下车的对象的多个相对位置的信息，确定所述多个相对位置中的目标相对位置，包括：
    响应于检测到的上下车的对象的相对位置信息包括表征多个上下车的对象的多个相对位置的信息，根据预先确定的不同位置信息的优先级顺序，将所述多个相对位置中优先级最高的相对位置确定为所述目标相对位置。
  7. 根据权利要求6所述的方法，其中，所述不同位置信息的优先级顺序包括以下至少之一：
    对应驾驶区域的位置信息的优先级，高于对应副驾驶区域的位置信息的优先级；
    对应前排区域的位置信息的优先级，高于对应后排区域的位置信息的优先级；
    对应专用座椅的位置信息的优先级，高于对应非专用座椅的位置信息的优先级。
  8. 根据权利要求1-7任一项所述的方法，在所述根据视频流中的图像，检测上下车的对象的相对位置信息之前，所述方法还包括：
    基于所述视频流中具备时序关系的多帧图像，确定对象的上下车意图；
    根据所述对象的上下车意图，确定所述上下车的对象。
  9. 根据权利要求8所述的方法，其中，所述基于所述视频流中具备时序关系的多帧图像，确定对象的上下车意图，包括以下至少之一：
    响应于在所述多帧图像中与车门距离小于预设距离的范围内检测到对象的至少一个部位，确定对象的下车意图；
    响应于在所述多帧图像中与车门距离小于预设距离的范围内未检测到对象的任何部位且所述车门处于开启状态，确定对象的上车意图。
  10. 根据权利要求1-9任一项所述的方法，其中，所述方法还包括：
    基于所述视频流中的图像，检测所述上下车的对象的属性信息；
    根据所述上下车的对象的属性信息，生成并显示所述数字人播报或展示与所述属性信息对应的多媒体信息的动画。
  11. 根据权利要求10所述的方法，其中，所述上下车的对象包括以下至少之一：人、宠物；
    所述对象的属性信息包括以下至少之一：身份信息、性别信息、类别信息；
    所述多媒体信息包括以下至少之一：表情信息、动作信息、语音信息。
  12. 一种基于车载数字人的交互装置，包括：
    获取模块，用于获取车内区域的视频流；
    相对位置检测模块，用于根据视频流中的图像，检测上下车的对象的相对位置信息，所述相对位置信息包括：所述上下车的对象在所述图像中的第一相对位置信息或者所述上下车的对象与车的第二相对位置信息；
    第一交互模块，用于根据所述相对位置信息，生成并显示所述车上设置的显示设备中显示的数字人执行相应转向的动画。
  13. 一种计算机可读存储介质，，所述计算机可读存储介质存储有计算机程序，当处理器执行所述计算机程序时，所述处理器用于执行上述权利要求1-11任一所述的基于车载数字人的交互方法。
  14. 一种基于车载数字人的交互装置，包括：
    处理器；
    用于存储所述处理器可执行指令的存储器；
    其中，所述处理器被配置为调用所述存储器中存储的可执行指令时，实现权利要求1-11中任一项所述的基于车载数字人的交互方法。
  15. 一种计算机程序产品，包括计算机可读代码，当所述计算机可读代码在处理器上运行时，所述处理器用于执行权利要求1-11中任一项所述的基于车载数字人的交互方法。
PCT/CN2020/136256 2020-06-23 2020-12-14 基于车载数字人的交互方法及装置、存储介质 Ceased WO2021258672A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP20941609.8A EP4170466A4 (en) 2020-06-23 2020-12-14 INTERACTION METHOD AND DEVICE BASED ON A VEHICLE-MOUNTED DIGITAL PERSON AND STORAGE MEDIUM
JP2022524647A JP2022553773A (ja) 2020-06-23 2020-12-14 車載デジタルヒューマンに基づくインタラクション方法及び装置、記憶媒体
KR1020217034787A KR20220000903A (ko) 2020-06-23 2020-12-14 차량 탑재형 디지털 인간에 기반한 상호작용 방법 및 장치, 저장 매체

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010581363.4A CN111736699A (zh) 2020-06-23 2020-06-23 基于车载数字人的交互方法及装置、存储介质
CN202010581363.4 2020-06-23

Publications (1)

Publication Number Publication Date
WO2021258672A1 true WO2021258672A1 (zh) 2021-12-30

Family

ID=72650632

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/136256 Ceased WO2021258672A1 (zh) 2020-06-23 2020-12-14 基于车载数字人的交互方法及装置、存储介质

Country Status (5)

Country Link
EP (1) EP4170466A4 (zh)
JP (1) JP2022553773A (zh)
KR (1) KR20220000903A (zh)
CN (1) CN111736699A (zh)
WO (1) WO2021258672A1 (zh)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111736699A (zh) * 2020-06-23 2020-10-02 上海商汤临港智能科技有限公司 基于车载数字人的交互方法及装置、存储介质
CN112832615A (zh) * 2021-01-04 2021-05-25 东风柳州汽车有限公司 辅助乘员下车的安全开门方法、装置、设备及存储介质
CN112379812B (zh) * 2021-01-07 2021-04-23 深圳追一科技有限公司 仿真3d数字人交互方法、装置、电子设备及存储介质
CN113674507A (zh) * 2021-08-18 2021-11-19 建信金融科技有限责任公司 婴儿监控方法、装置、计算机设备及计算机可读存储介质
CN116841482A (zh) * 2022-03-23 2023-10-03 华人运通(上海)云计算科技有限公司 一种车辆交互屏控制方法、装置、车辆及存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108664123A (zh) * 2017-12-15 2018-10-16 蔚来汽车有限公司 人车交互方法、装置、车载智能控制器及系统
CN109366497A (zh) * 2018-11-12 2019-02-22 奇瑞汽车股份有限公司 车载机器人、车载机器人的控制方法、装置及存储介质
CN110502116A (zh) * 2019-08-20 2019-11-26 广东远峰汽车电子有限公司 汽车情感机器人与乘车人员的互动方法及装置
CN110641476A (zh) * 2019-08-16 2020-01-03 广汽蔚来新能源汽车科技有限公司 基于车载机器人的交互方法、装置、控制器和存储介质
CN110682921A (zh) * 2019-10-09 2020-01-14 广州小鹏汽车科技有限公司 车辆交互方法、装置、车辆及机器可读介质
US20200039427A1 (en) * 2017-10-31 2020-02-06 Tencent Technology (Shenzhen) Company Limited Interaction method and apparatus of mobile robot, mobile robot, and storage medium
CN111736699A (zh) * 2020-06-23 2020-10-02 上海商汤临港智能科技有限公司 基于车载数字人的交互方法及装置、存储介质

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008250453A (ja) * 2007-03-29 2008-10-16 Aisin Aw Co Ltd 運転支援装置および運転支援方法
JPWO2009107185A1 (ja) * 2008-02-25 2011-06-30 パイオニア株式会社 車載ロボット
JP5988472B2 (ja) * 2011-07-20 2016-09-07 株式会社日立国際電気 監視システム、および、混雑率算出方法
JP2019156237A (ja) * 2018-03-14 2019-09-19 三菱自動車工業株式会社 注意喚起システム
JP7053432B2 (ja) * 2018-11-14 2022-04-12 本田技研工業株式会社 制御装置、エージェント装置及びプログラム

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200039427A1 (en) * 2017-10-31 2020-02-06 Tencent Technology (Shenzhen) Company Limited Interaction method and apparatus of mobile robot, mobile robot, and storage medium
CN108664123A (zh) * 2017-12-15 2018-10-16 蔚来汽车有限公司 人车交互方法、装置、车载智能控制器及系统
CN109366497A (zh) * 2018-11-12 2019-02-22 奇瑞汽车股份有限公司 车载机器人、车载机器人的控制方法、装置及存储介质
CN110641476A (zh) * 2019-08-16 2020-01-03 广汽蔚来新能源汽车科技有限公司 基于车载机器人的交互方法、装置、控制器和存储介质
CN110502116A (zh) * 2019-08-20 2019-11-26 广东远峰汽车电子有限公司 汽车情感机器人与乘车人员的互动方法及装置
CN110682921A (zh) * 2019-10-09 2020-01-14 广州小鹏汽车科技有限公司 车辆交互方法、装置、车辆及机器可读介质
CN111736699A (zh) * 2020-06-23 2020-10-02 上海商汤临港智能科技有限公司 基于车载数字人的交互方法及装置、存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4170466A4 *

Also Published As

Publication number Publication date
KR20220000903A (ko) 2022-01-04
CN111736699A (zh) 2020-10-02
EP4170466A1 (en) 2023-04-26
JP2022553773A (ja) 2022-12-26
EP4170466A4 (en) 2023-12-06

Similar Documents

Publication Publication Date Title
WO2021258672A1 (zh) 基于车载数字人的交互方法及装置、存储介质
US20220189093A1 (en) Interaction based on in-vehicle digital persons
US9942522B2 (en) In-vehicle camera system
US9865258B2 (en) Method for recognizing a voice context for a voice control function, method for ascertaining a voice control signal for a voice control function, and apparatus for executing the method
US9928404B2 (en) Determination device, determination method, and non-transitory storage medium
JP2022176081A (ja) 適応視標追跡機械学習モデル・エンジン
JP2022530605A (ja) 子供状態検出方法及び装置、電子機器、記憶媒体
US10747007B2 (en) Intelligent vehicle point of focus communication
CN109849788B (zh) 信息提供方法、装置及系统
JP7469467B2 (ja) デジタルヒューマンに基づく車室インタラクション方法、装置及び車両
CN110877581A (zh) 儿童乘车看护方法、装置、车辆及存储介质
JP2022071801A (ja) 情報処理装置及び情報処理方法
Rong et al. Artificial intelligence methods in in-cabin use cases: A survey
WO2021258671A1 (zh) 基于车载数字人的辅助驾驶交互方法及装置、存储介质
CN111736700A (zh) 基于数字人的车舱交互方法、装置及车辆
Meng et al. Application and development of AI technology in automobile intelligent cockpit
CN112836549A (zh) 用户信息的检测方法及系统、电子设备
CN118418994A (zh) 泊车方法、装置、车辆及计算机可读取存储介质
JP7537259B2 (ja) 注目対象共有装置、注目対象共有方法
WO2022239642A1 (ja) 車両用情報提供装置、車両用情報提供方法、及び車両用情報提供プログラム
US20250303279A1 (en) Game Control System and Game Control Method Based on Visual Recognition Algorithm
JP7838640B2 (ja) 情報提供方法及び情報提供システム
WO2026037366A1 (zh) 吹风控制方法、装置和车辆
CN116931850A (zh) 车载显示屏控制方法、装置、介质及设备
CN117234444A (zh) 智能机器人屏幕形象的显示方法、装置、设备和介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20941609

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022524647

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020941609

Country of ref document: EP

Effective date: 20230123