JP2020006759A - Extractor, control system, communication system, learning model, information processor, computer program and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an extracting device, a control system, a communication system, a learning model, an information processing device, a computer program, and a storage medium capable of efficiently extracting information on a traveling environment. An extraction device 1 is, for example, a computer mounted on a human-powered vehicle, and extracts information about a traveling environment of the human-powered vehicle by a learning algorithm from a detection result of a surrounding environment of the human-powered vehicle. The information on the traveling environment of the human-powered vehicle includes information on the obstacle, and the obstacle includes, for example, a step extending along the traveling of the human-driven vehicle. The output unit 14 includes an output interface that outputs the extraction result of the arithmetic processing unit 12. The output unit 14 may output an image showing the extraction result obtained from the arithmetic processing unit 12. [Selection diagram] Figure 2

Description

  The present invention relates to an extraction device, a control system, a communication system, a learning model, an information processing device, a computer program, and a storage medium.

  2. Description of the Related Art In an automobile, an obstacle detection device that detects the presence or absence of an obstacle in front of the vehicle is known (for example, see Patent Document 1).

JP 2006-12356 A

  However, Patent Document 1 assumes obstacle detection in a vehicle that uses an internal combustion engine or an electric motor as a drive source, and does not propose a method for efficiently extracting information about a traveling environment in a manually driven vehicle.

  An object of the present invention is to provide an extraction device, a control system, a communication system, a learning model, an information processing device, a computer program, and a storage medium that can efficiently extract information about a driving environment.

  [1] An extraction device according to one aspect of the present invention includes an extraction unit that extracts information on a traveling environment of a manually driven vehicle from a detection result of a surrounding environment of the manually driven vehicle using a learning algorithm.

  According to this extraction device, since a learning algorithm for estimating the traveling environment from the detection result of the surrounding environment of the manually driven vehicle by learning such as machine learning and deep learning is used, information on the traveling environment can be efficiently extracted. The detection result of the surrounding environment of the manually driven vehicle is, for example, image data obtained by imaging the surroundings of the manually driven vehicle by an imaging device, distance data from the manually driven vehicle to an obstacle measured using a distance measurement sensor. And so on. The information on the traveling environment extracted by the extraction device includes information on obstacles. The information on the obstacle is, for example, information on a step extending along the traveling of the manually driven vehicle. The information on the obstacle may be information on unevenness or slipperiness of the road surface, or information on an object existing on the road surface.

  [2] In the extraction device according to an aspect of the present invention, the learning algorithm includes a deep learning algorithm.

  According to this extraction device, since a deep learning algorithm for estimating a traveling environment from a detection result of a surrounding environment of a manually driven vehicle by deep learning is used, a process of extracting a feature amount indicating a feature of the detection result becomes unnecessary, and It is possible to accurately extract information on the driving environment without being influenced by the result of the extraction of the amount.

  [3] In the extraction device according to an aspect of the present invention, the detection result includes an image around the manually driven vehicle.

  According to this extraction device, information about the driving environment can be efficiently extracted by analyzing the image around the manually driven vehicle using the learning algorithm.

  [4] In the extraction device according to an aspect of the present invention, the information on the traveling environment includes information on an obstacle.

  According to this extraction device, information on obstacles can be efficiently extracted by analyzing the detection result of the surrounding environment of the manually driven vehicle using the learning algorithm.

  [5] In the extraction device according to an aspect of the present invention, the information on the obstacle includes information on a step extending along a traveling direction of the manually driven vehicle.

  According to this extraction device, information on a step extending along the traveling direction of the manually driven vehicle can be extracted by analyzing the detection result of the surrounding environment of the manually driven vehicle using the learning algorithm.

  [6] A control system according to one aspect of the present invention includes the extraction device described above and a control device that controls components of the manually driven vehicle based on an extraction result of the extraction device.

  According to this control system, since the information on the traveling environment of the manually driven vehicle is extracted by the extraction device, the components of the manually driven vehicle can be controlled according to the traveling environment extracted by the extraction device.

  [7] In the control system according to an aspect of the present invention, the component includes a notification device.

  According to this control system, since the component to be controlled includes the notification device, the information on the traveling environment extracted by the extraction device can be notified to the passenger of the manually driven vehicle.

  [8] In the control system according to an aspect of the present invention, the notification device includes a lighting device.

  According to this control system, the notification device includes the lighting device. For example, when the information on the driving environment includes an obstacle or the like, the information on the obstacle or the like is manually input by illuminating the obstacle or the like. It can notify the passenger of the driving car.

  [9] In the control system according to an aspect of the present invention, the component includes a support device for a steering device of the manually driven vehicle.

  According to this control system, since information about the traveling environment of the manually driven vehicle is obtained by the extraction device, the operability of the manually driven vehicle can be improved by supporting the steering of the manually driven vehicle according to the traveling environment.

  [10] A communication system according to an aspect of the present invention includes the above-described extraction device and a communication device including a communication unit that communicates with an external device. The communication unit transmits the extraction result of the extraction device to the external device. Send to device.

  According to this communication system, communication with an external device can be performed via the communication device, so that expandability is improved.

  [11] The communication system according to an aspect of the present invention, wherein the communication device further includes a position information acquisition unit that acquires position information, wherein the communication unit acquires the position information acquired by the position information acquisition unit, and The extraction result of the extraction device is transmitted to an external device.

  According to this communication system, the position information of the manually driven vehicle and the extraction result extracted by the extraction device are transmitted to the external device, so that the location where the extraction result is obtained can be grasped by the external device.

  [12] In the communication system according to an aspect of the present invention, the external device generates information on a map from the position information from the communication device and the extraction result.

  According to this communication system, it is possible to generate a map indicating the location where the extraction result is obtained.

  [13] In the communication system according to an aspect of the present invention, the communication device receives, from the external device, position information of another manually driven vehicle and an extraction result of another manually driven vehicle.

  According to this communication system, the extraction result can be shared among a plurality of manually driven vehicles.

  [14] The learning model according to one aspect of the present invention outputs an input layer to which a detection result of the surrounding environment of the manually driven vehicle is input and an estimation result indicating whether the detection result includes an obstacle. An output layer, an intermediate layer based on teacher data including a detection result including an obstacle and label data indicating that the detection result includes an obstacle, and detecting the surrounding environment input to the input layer. Based on the result, the calculation is performed in the hidden layer, and is used for a process of outputting an estimation result regarding the detection result from the output layer.

  According to this learning model, a computer can realize an execution environment that outputs an estimation result indicating whether an obstacle is included or not from a detection result of the surrounding environment of the manually driven vehicle.

  [15] In the learning model according to one aspect of the present invention, the obstacle includes a step extending along a traveling direction of the manually driven vehicle.

  According to this learning model, a computer can realize an execution environment that outputs an estimation result indicating whether or not a step extending along the traveling direction of the manually driven vehicle is included, based on the detection result of the surrounding environment of the manually driven vehicle.

  [16] In the learning model according to an aspect of the present invention, the detection result includes an image around the manually driven vehicle.

  According to this learning model, a computer can realize an execution environment for outputting an estimation result of an obstacle including a step extending in the traveling direction of the manually driven vehicle from an image around the manually driven vehicle.

  [17] In the learning model according to one aspect of the present invention, the teacher data includes a detection result that does not include an obstacle and label data indicating that the detection result does not include an obstacle.

  According to this learning model, teacher data including a detection result including an obstacle, label data indicating that the detection result includes an obstacle, a detection result including no obstacle, and a detection result including an obstacle Based on both the teacher data and the label data indicating that the object is not included, the estimation accuracy can be improved.

  [18] In the learning model according to one aspect of the present invention, the teacher data includes travel information detected during travel of the manually driven vehicle.

  According to this learning model, it is determined whether or not the detection result of the surrounding environment of the manually driven vehicle includes an obstacle based on the traveling information detected during the traveling of the manually driven vehicle, and the determination result is used as teacher data. Can be included.

  [19] An information processing apparatus according to an aspect of the present invention, wherein a calculation result of inputting a detection result about a surrounding environment of a manually driven vehicle to the learning model and acquiring an estimation result regarding the detection result from the learning model is obtained. It has a unit.

  According to this information processing device, by inputting the detection result about the surrounding environment of the manually driven vehicle to the learning model, the estimation result about whether the detection result includes an obstacle can be obtained, and thus the driving environment can be efficiently performed. You can get information about

  [20] The information processing apparatus according to one aspect of the present invention includes a receiving unit that receives an updated learning model updated by additional learning based on a detection result of a surrounding environment of the manually driven vehicle, wherein the updated learning model When receiving the selection to update to the model, the learning model is updated to the updated learning model.

  According to this information processing apparatus, an updated learning model updated by additional learning can be received and updated as necessary. Therefore, it is not necessary to perform the additional learning in the information processing apparatus, and the external server or the like having a higher processing capability can execute the additional learning and receive the updated learning model updated from the external server or the like. Can reduce the processing load.

  [21] The information processing device according to an aspect of the present invention, wherein the obstacle estimated by the learning model is a step extending along the traveling direction of the manually driven vehicle, and the detection result includes the obstacle. The arithmetic processing unit acquires the estimated result shown, the distance between the manually driven vehicle and the step is less than a set distance, and the angle of entry of the manually driven vehicle into the step is less than a set angle. In this case, an output unit for outputting warning information is provided.

  According to this information processing device, an estimation result for a step extending along the traveling direction of the manually driven vehicle is obtained, and the distance between the step and the manually driven vehicle is less than the set distance, and the approach angle to the step Is smaller than the set angle, the warning information is output, so that it is possible to reduce the possibility that traveling by the manually driven vehicle becomes difficult.

  [22] A computer program according to an aspect of the present invention causes a computer to execute a process of extracting information on a traveling environment of a manually driven vehicle from a detection result of a surrounding environment of the manually driven vehicle by a learning algorithm. Computer program.

  According to this computer program, a computer can realize an execution environment for extracting information on a traveling environment of a manually driven vehicle from a detection result of a surrounding environment of the manually driven vehicle using a learning algorithm.

  [23] A storage medium according to one embodiment of the present invention stores the above-described computer program.

  According to this storage medium, by installing a computer program stored in the storage medium into a computer, information on a traveling environment of the manually driven vehicle is obtained from a detection result of a surrounding environment of the manually driven vehicle using a learning algorithm. Can be realized.

  According to the present application, information on the traveling environment of the manually driven vehicle can be efficiently extracted from the detection result of the surrounding environment of the manually driven vehicle.

FIG. 2 is a schematic explanatory diagram illustrating an extraction device according to the first embodiment. It is a block diagram showing the internal configuration of the extraction device concerning a 1st embodiment. FIG. 2 is a schematic diagram illustrating a configuration example of a learning model according to the first embodiment. It is a flowchart explaining the procedure of the process which the extraction device which concerns on 1st Embodiment performs. FIG. 3 is a schematic diagram illustrating an example of an input image. It is a schematic diagram which shows an example of an output image. It is a block diagram explaining a control system concerning a 2nd embodiment. It is a flowchart which shows the control procedure in 2nd Embodiment. FIG. 13 is a block diagram illustrating a communication system according to a third embodiment. FIG. 3 is a block diagram illustrating an internal configuration of a server device. It is a flowchart which shows the control procedure of the communication apparatus in 3rd Embodiment. It is a flowchart which shows the control procedure of the server apparatus in 3rd Embodiment. FIG. 2 is a block diagram illustrating an internal configuration of the information processing apparatus. It is a flowchart explaining an update procedure. It is a flowchart explaining an update procedure.

Hereinafter, the present invention will be specifically described with reference to the drawings showing the embodiments.
(1st Embodiment)
FIG. 1 is a schematic explanatory diagram illustrating an extraction device 1 according to the first embodiment. The extraction device 1 according to the first embodiment is, for example, a computer mounted on the human-powered vehicle 100. Based on a detection result of the surrounding environment of the human-powered vehicle 100, information on the traveling environment of the human-powered vehicle 100 is obtained by a learning algorithm. Extract. In the present embodiment, the human-powered vehicle 100 is a vehicle that at least partially uses human power for driving power for traveling, and includes a vehicle that electrically assists human power. Vehicles using only an internal combustion engine or an electric motor as a driving force are excluded from the manually driven vehicle 100. In the present embodiment, the manually driven vehicle 100 is a bicycle. The manually driven vehicle 100 includes a steering device H. One example of the steering device H is a handlebar. The extraction device 1 may be a dedicated terminal provided in the manually driven vehicle 100, or may be a mobile terminal such as a smartphone carried by a passenger of the manually driven vehicle 100.

  The detection result on the surrounding environment of the manually driven vehicle 100 includes an image around the manually driven vehicle 100. In the present embodiment, the detection result of the surrounding environment of the manually driven vehicle 100 is, for example, an image obtained by capturing an image of the periphery of the manually driven vehicle 100 by an imaging device P such as a digital camera. The detection result of the surrounding environment acquired by the extraction device 1 may be a distance from the manually driven vehicle 100 to an obstacle, which is measured using a distance measuring sensor such as an ultrasonic sensor.

  The extraction device 1 extracts information on the traveling environment of the manually driven vehicle 100 from the acquired detection result by a learning algorithm. The learning algorithm is an algorithm for estimating the traveling environment of the manually driven vehicle 100 from the detection result of the surrounding environment of the manually driven vehicle 100 using the learning model 130 illustrated in FIG. The learning algorithm includes a learning algorithm such as a machine learning algorithm based on machine learning and a deep learning algorithm based on deep learning.

  The information on the traveling environment of the manually driven vehicle 100 includes information on obstacles. The obstacle is an object that hinders traveling. The obstacle includes, for example, a step S extending along the traveling traveling of the manually driven vehicle 100. Such a step S exists, for example, between a roadway and a sidewalk. The step S shown in FIG. 1 exists on the left side with respect to the traveling direction of the manually driven vehicle 100. Here, front, rear, left and right represent front, rear, left, and right as viewed from a state where the occupant of the manually driven vehicle 100 straddles the saddle toward the steering wheel.

  The step S extending along the traveling direction of the manually driven vehicle 100 extends substantially parallel to, for example, the traveling direction of the manually driven vehicle 100, or has an angle range of, for example, less than 10 degrees with respect to the traveling direction of the manually driven vehicle 100. Extends diagonally to the left or right within. If the entry angle of the manually driven vehicle 100 when climbing over the step S is a relatively shallow angle of, for example, less than 10 degrees, traveling may be difficult. For this reason, the extraction device 1 extracts information on the step S extending along the traveling direction of the manually driven vehicle 100 from the detection result of the surrounding environment as an extraction target. Further, among the steps S extending along the traveling direction of the manually driven vehicle 100, those having a height lower than the set height are set as extraction targets, and those having a height higher than the set height are excluded from the extraction targets. Good. The set height is, for example, 20 cm, but can be set as appropriate depending on the type of the manually driven vehicle 100, the size of the wheels included in the manually driven vehicle 100, and the like.

  In addition, the information on the obstacle may include information on a gutter, a manhole, unevenness of a road surface, a slippery road surface, and an object (vehicle, person, animal, etc.) existing around the manually driven vehicle 100.

  Hereinafter, a configuration will be described in which information on a step S extending along the traveling direction of the bicycle is extracted from an image captured by the imaging device P by a learning algorithm.

  FIG. 2 is a block diagram showing the internal configuration of the extraction device 1 according to the first embodiment. The extraction device 1 includes an extraction unit 10 including an input unit 11, an arithmetic processing unit 12, a storage unit 13, and an output unit 14. The extraction unit 10 uses these hardware units to extract information on the running environment of the bicycle from the detection result on the surrounding environment of the bicycle by a learning algorithm.

  The input unit 11 includes an input interface for connecting the imaging device P. Image data of an image output from the imaging device P is input to the input unit 11. In the present embodiment, the imaging device P has an angle of view suitable for extracting information of the step S, and is attached to a position suitable for imaging the front in the traveling direction in the main body of the manually driven vehicle 100. Is preferred. The image data output from the imaging device P is still image data or moving image data. The image data input to the input unit 11 is, for example, frame-based data composed of a plurality of pixels, each pixel being represented by a gradation value of each of RGB colors. The input unit 11 outputs the input image data to the arithmetic processing unit 12.

  The arithmetic processing unit 12 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The ROM included in the arithmetic processing unit 12 stores a computer program or the like for controlling the operation of each unit of the hardware included in the extraction device 1. The CPU in the arithmetic processing unit 12 executes a computer program stored in the ROM, controls the operation of each unit of the hardware, and realizes a process of extracting information about the driving environment from the input image data. Data used during execution of the operation is temporarily stored in the RAM included in the operation processing unit 12.

  Although the arithmetic processing unit 12 has a configuration including a CPU, a ROM, and a RAM, a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a quantum processor, a volatile or nonvolatile memory It may be one or a plurality of arithmetic circuits including the above.

  The storage unit 13 includes memories such as an EEPROM (Electronically Erasable Programmable Read Only Memory) and an SRAM (Static Random Access Memory). The storage unit 13 stores a computer program executed by the arithmetic processing unit 12, a learning model 130, and the like.

  The storage unit 13 stores an estimation processing program. The estimation processing program is a computer program for causing a computer to execute a process of extracting information on a traveling environment of the manually driven vehicle 100 from a detection result of a surrounding environment of the manually driven vehicle 100 by a learning algorithm. By executing the estimation processing program, the arithmetic processing unit 12 extracts information on the traveling environment from the detection result of the surrounding environment of the manually driven vehicle 100 based on a learning algorithm for estimating the traveling environment using the learning model 130.

  A computer program including the estimation processing program can be provided by a storage medium M in which the computer program is stored. The storage medium M is, for example, a portable memory such as a CD-ROM, a USB memory, an SD (Secure Digital) card, a micro SD card, and a compact flash (registered trademark). In the present embodiment, the storage medium M is a non-transitory storage medium that stores a computer program in a readable manner. The arithmetic processing unit 12 reads various programs from the storage medium M using a reading device, and installs the read various programs in the storage unit 13.

  The learning model 130 stored in the storage unit 13 is described by the definition information. The definition information of the learning model 130 includes structural information such as a neural network constituting the learning model 130, various parameters used in the learning model 130, and the like. Various parameters used in the learning model 130 include, for example, weights and biases between nodes constituting the neural network. In the present embodiment, the learning unit 130 learns in advance the learning model 130 based on the teacher data including the image data including the step S to be extracted and the label data indicating that the image data includes the step S to be extracted. Is stored.

  The output unit 14 includes an output interface that outputs a result of the extraction performed by the arithmetic processing unit 12. The output format of the extraction result is arbitrary. The output unit 14 may output an image indicating the extraction result obtained from the arithmetic processing unit 12. Further, the output unit 14 may output character information indicating that the information on the step S has been extracted based on the extraction result of the arithmetic processing unit 12. Furthermore, when the information of the step S is extracted by the arithmetic processing unit 12, the output unit 14 may output a sound or a voice that prompts the passenger to call attention. When the extraction device 1 includes a user interface such as an operation button, the extraction device 1 may receive a selection of an output format by a passenger. In this case, the output unit 14 may output the information according to the output format selected by the passenger.

  FIG. 3 is a schematic diagram illustrating a configuration example of the learning model 130 according to the first embodiment. The learning model 130 includes an input layer 131, an intermediate layer 132, and an output layer 133. An example of a learning algorithm determined by the learning model 130 is, for example, an algorithm of R-CNN (Region-based Convolutional Neural Networks).

  The detection result of the surrounding environment of the manually driven vehicle 100 is input to the input layer 131. In the present embodiment, the detection result of the surrounding environment of the manually driven vehicle 100 includes an image around the manually driven vehicle 100. When inputting image data relating to an image around the human-powered vehicle 100 to the input layer 131, a plurality of partial regions including some objects included in the image are extracted, and all the extracted partial regions are resized to a certain size. And input to the input layer 131. The objects included in each partial area may include not only the step S to be extracted, but also any object such as a person or another vehicle. For the extraction of the region, for example, a method disclosed in “Selective search for object recognition.”, Uijlings, Jasper RR, et al. International journal of computer vision 104.2 (2013), 154-171 is used.

  The intermediate layer 132 is based on teacher data including a detection result including an obstacle and label data indicating that the detection result includes an obstacle. In the present embodiment, the detection result including the obstacle is image data including the step S to be extracted, and the label data is data indicating that the image data includes the step S to be extracted. The teacher data may include a detection result that does not include an obstacle and label data indicating that the detection result does not include an obstacle. The intermediate layer 132 includes, for example, a convolutional layer 132a, a pooling layer 132b, and a total connection layer 132c. A plurality of convolutional layers 132a and pooling layers 132b in the intermediate layer 132 may be provided alternately. The convolutional layer 132a and the pooling layer 132b extract image feature values from the area image of each partial area input through the input layer 131. The feature amounts extracted by the convolution layer 132a and the pooling layer 132b are, for example, a brightness difference, a luminance gradient, a shape, an edge, a texture, and the like. The features extracted by the convolutional layer 132a and the pooling layer 132b are output to the fully connected layer 132c. The fully connected layer 132c classifies the objects in each partial area based on the feature amount of the image input from the final pooling layer 132b, and outputs the classification result to the output layer 133.

  The output layer 133 outputs an estimation result as to whether or not the detection result of the surrounding environment of the manually driven vehicle 100 input to the input layer 131 includes an obstacle, based on the classification result input from the total connection layer 132c. In the present embodiment, the obstacle is a step S extending along the traveling direction of the manually driven vehicle 100. The estimation result output from the output layer 133 may include probability information indicating the likelihood of the estimation result. The estimation result output from the output layer 133 may include information on a region surrounding the estimated object. The area surrounding the estimated object is, for example, a rectangular area, and is specified by the coordinates of the center position, the width and the height, or the coordinates of the four corners.

  FIG. 4 is a flowchart illustrating a procedure of a process performed by the extraction device 1 according to the first embodiment. In step S101, the arithmetic processing unit 12 of the extraction device 1 acquires a detection result of the surrounding environment. Specifically, the arithmetic processing unit 12 acquires image data obtained by the imaging device P through the input unit 11.

  Next, in step S <b> 102, the calculation processing unit 12 executes a calculation by a learning algorithm by inputting the acquired image data to the input layer 131 of the learning model 130. At this time, the arithmetic processing unit 12 refers to the definition information of the learning model 130 stored in the storage unit 13. Further, the arithmetic processing unit 12 executes an estimation processing program stored in the storage unit 13 to execute an arithmetic operation on the input image data by a learning algorithm for estimating a driving environment using the learning model 130.

  Next, the arithmetic processing unit 12 acquires, from the output layer 133, information on the driving environment extracted by the learning model 130 in step S103. That is, the learning model 130 is used in a process of calculating in the intermediate layer 132 based on the detection result of the surrounding environment input to the input layer 131 and outputting an output result related to the detection result from the output layer 133. In addition, the arithmetic processing unit 12 inputs a detection result regarding the surrounding environment of the manually driven vehicle 100 to the learning model 130, and acquires an estimation result regarding the detection result from the learning model 130. Information obtained from the output layer 133 of the learning model 130 includes an estimation result of whether or not the detection result of the surrounding environment of the manually driven vehicle 100 input to the input layer 131 includes an obstacle.

  Further, the arithmetic processing unit 12 may generate an output image to be notified to the passenger based on the information acquired from the learning model 130. FIG. 5A is a schematic diagram illustrating an example of an input image, and FIG. 5B is a schematic diagram illustrating an example of an output image. FIG. 5A illustrates an example of an input image obtained by capturing an image of the front of the human-powered vehicle 100 in the traveling direction using the imaging device P. The input image shown in FIG. 5A includes a step S on the right side and the left side with respect to the traveling direction of the manually driven vehicle 100.

  FIG. 5B illustrates an example of an output image generated by the arithmetic processing unit 12 based on information acquired from the learning model 130. When the estimation result that the step S is included in the specific partial region in the input image is obtained, and the probability indicating the certainty of the estimation result is equal to or more than the threshold, the arithmetic processing unit 12 performs the processing shown in FIG. An output image may be generated. The threshold for the probability can be set as appropriate. One example of the threshold is 80%.

  The arithmetic processing unit 12 can specify the position of the step S in the image based on the information of the area obtained from the learning model 130. The arithmetic processing unit 12 can generate an image in which the step S is emphasized as shown in FIG. 5B by increasing the density of each pixel in the area including the step S and decreasing the density of each pixel outside the area. The arithmetic processing unit 12 may generate an image in which the step S is emphasized by changing the color of each pixel in the area including the step S to a conspicuous color such as red. The arithmetic processing unit 12 may generate an output image with a marker indicating the position of the step S and character information indicating a warning. In the example of FIG. 5B, an output image in which a marker M1 and character information N1 are added to a step S existing on the left side in the traveling direction, and a marker M2 and character information N2 are added to the step S existing on the right side. Is shown.

  As described above, in the first embodiment, the information of the step S can be efficiently extracted from the captured image obtained while the manually driven vehicle 100 is traveling by using the learning algorithm.

  In the present embodiment, the configuration in which the information of the step S to be extracted is extracted using R-CNN, which is one of the deep learning algorithms, has been described. However, the deep learning algorithm used in the learning model 130 is R-CNN. It is not limited to. For example, instead of R-CNN, a deep learning algorithm including SPPnet (Spatial Pyramid Pooling network), Fast R-CNN, Faster R-CNN, YOLO (You Only Look Once), SSD (Single Shot multibox Detector) and the like is used. You may. In addition, as a learning algorithm other than the deep learning, a learning algorithm such as a support vector machine or a Bayesian classification may be used to extract information on a step to be extracted.

(2nd Embodiment)
In the second embodiment, a control system that controls the component C of the manually driven vehicle 100 based on the extraction result by the extraction device 1 will be described.

  FIG. 6 is a block diagram illustrating a control system according to the second embodiment. The control system according to the second embodiment includes an extraction device 1 and a control device 2 that controls a component C of the manually driven vehicle 100 based on an extraction result by the extraction device 1. Since the internal configuration of the extraction device 1 is the same as that of the first embodiment, a detailed description thereof will be omitted.

  The control device 2 includes a control unit 21, a storage unit 22, an input unit 23, and an output unit 24. The control unit 21 includes, for example, a CPU, a ROM, a RAM, and the like. The ROM included in the control unit 21 stores a control program and the like for controlling the operation of each unit of the hardware included in the control device 2. The CPU in the control unit 21 executes a control program stored in the ROM and various programs stored in the storage unit 22 to control the operation of each unit of the hardware, thereby realizing the control device of the present application. Specifically, the control unit 21 generates a control signal for controlling the component C of the manually driven vehicle 100 based on the extraction result from the extraction device 1 input through the input unit 23, and outputs the generated control signal to the output unit. 24.

  The control unit 21 is not limited to the configuration described above. The control unit 21 may be any one or a plurality of control circuits including a single-core CPU, a multi-core CPU, an FPGA, a volatile or nonvolatile memory, and the like. In addition, the control unit 21 may include a clock for outputting date and time information, a timer for measuring an elapsed time from when a measurement start instruction is given to when a measurement end instruction is given, and a counter for counting the number.

  The storage unit 22 includes a memory such as an EEPROM and an SRAM. The storage unit 22 stores a computer program executed by the control unit 21, data used by the computer program, and the like.

  The input unit 23 includes an interface for connecting the extraction device 1 via the signal line L1. The input unit 23 receives an extraction result output from the extraction device 1. The extraction result input to the input unit 23 may be image data indicating the extraction result, or may be character information or a signal indicating the extraction result. The input unit 23 outputs the input extraction result to the control unit 21. In the present embodiment, the extraction device 1 is connected via the signal line L1. However, it is configured such that data can be transmitted and received between the extraction device 1 and the control device 2 via the wireless communication interface. Is also good. In wireless communication, a wireless transmission method conforming to communication standards such as Bluetooth (registered trademark), WiFi (registered trademark), ZigBee (registered trademark), and other wireless LAN (Local Area Network) can be used.

  The output unit 24 includes an interface that connects the component C of the manually driven vehicle 100 to be controlled via the signal line L2. The output unit 24 acquires the control signal generated by the control unit 21 and outputs the acquired control signal to the component C to be controlled. In the present embodiment, the component C of the manually driven vehicle 100 is connected via the signal line L2. However, data is transmitted between the control device 2 and the component C of the manually driven vehicle 100 via a wireless communication interface. May be configured to be transmitted and received. In wireless communication, a wireless transmission system conforming to the various communication standards described above can be used.

  The component C of the bicycle includes the notification device N. The notification device N includes a lighting device 210. In the present embodiment, the notification device N further includes a display device 230 and a sound output device 220. The component C of the bicycle includes the assist device 240 of the steering device H of the manually driven vehicle 100.

  The lighting device 210 is a headlight that illuminates the traveling direction of the manually driven vehicle 100, for example. The lighting device 210 is configured to turn on, blink, or turn off according to a control signal output from the output unit 24. When the information of the step S is extracted by the extraction device 1, the control device 2 turns on or blinks the lighting device 210 for a set time to urge the passenger to be alert.

  The display device 230 includes a liquid crystal display panel or the like that displays information based on a control signal output from the output unit 24. The display device 210 is installed at an appropriate position of the manually driven vehicle 100 that can be visually recognized by a passenger. When the step S is extracted by the extraction device 1, the control device 2 displays an image for highlighting the step S, a marker indicating the position of the step S, an image including character information indicating a warning, and the like on the display device 230. By displaying the message, the passenger is alerted.

  The audio output device 220 includes a speaker or the like that outputs audio based on a control signal output from the output unit 24. The sound output device 220 is installed at an appropriate position of the human-powered vehicle 100 where the sound to be output can be heard by a passenger. When the step S is extracted by the extraction device 1, the control device 2 causes the sound output device 220 to output a sound indicating that the step S exists, thereby urging the passenger to be alert. The sound output device 220 is not limited to sound, and may be configured to output a warning sound.

  In the present embodiment, the illumination device 210, the display device 230, and the audio output device 220 have been described as examples of the notification device, but the notification device is not limited to these devices. For example, vibration devices may be provided on the left and right handles. When it is determined based on the extraction result from the extraction device 1 that there is a step S on the left side in the traveling direction, the control device 2 may output a control signal for operating a vibration device provided on the left handle. The control device 2 can notify the occupant of the presence of the step S on the left side in the traveling direction by operating the vibration device provided on the left handle. The same applies to the case where it can be determined that there is a step S on the right side in the traveling direction.

  The support device 240 includes at least a support mechanism for supporting the steering of the steering device H of the manually driven vehicle 100, an actuator of the support mechanism, and a driver. The assist device 240 is configured so that the driver operates the assist mechanism based on a control signal output from the output unit 24 with an actuator to assist the steering of the manually driven vehicle 100.

  When the step S is extracted by the extraction device 1, the control device 2 determines whether the step S exists on the right side or the left side with respect to the traveling direction of the manually driven vehicle 100. If the control device 2 determines that the step S exists on the right side in the traveling direction, the control device 2 outputs to the support device 240 a control signal for controlling the traveling direction of the manually driven vehicle 100 to be bent rightward. The support device 240 drives the actuator in accordance with the control signal from the control device 2 and supports the steering of the manually driven vehicle 100 so that the traveling direction is turned rightward. By performing such control, the entry angle of the manually driven vehicle 100 with respect to the step S is increased, and the possibility that the traveling by the manually driven vehicle 100 becomes difficult due to the shallow entry angle of the step S is reduced. Can be. The same applies when it is determined that the step S exists on the left side in the traveling direction, and the control device 2 assists the steering of the manually driven vehicle 100 so that the traveling direction is bent leftward.

  FIG. 7 is a flowchart illustrating a control procedure according to the second embodiment. The control unit 21 of the control device 2 executes the following procedure each time the extraction result by the extraction device 1 is obtained. The control part 21 acquires the extraction result by the extraction device 1 in step S201. In step S202, the control unit 21 determines whether or not the step S extracted by the extraction device 1 exists based on the extraction result obtained in step S201. The information of the step S extracted by the extraction device 1 according to the second embodiment is information of the step S that extends along the traveling direction of the manually driven vehicle 100 and whose height is lower than the set height, and the approach angle is When the vehicle is relatively shallow, the level difference S indicates that traveling by the manually driven vehicle 100 may be difficult. When the control unit 21 determines that the step S extracted by the extraction device 1 does not exist in step S202, the process according to this flowchart ends.

  When determining that the step S extracted by the extraction device 1 exists in step S202, the control unit 21 calculates the distance from the manually driven vehicle 100 to the step S in step S203. The control unit 21 can acquire an input image to the learning model 130 from the extraction device 1 and calculate a distance based on the acquired input image. First, the control unit 21 derives a linear formula that overlaps the step S and a linear formula that extends the traveling direction of the manually driven vehicle 100 in the input image. The control unit 21 can calculate the distance from the manually driven vehicle 100 to the step S from the distance between the coordinates of the point at which the two derived linear equations intersect and the coordinates indicating the ground contact point of the tire. Since the imaging device P is fixed, the coordinates indicating the ground contact point of the tire may be coordinates of a preset point. When a distance measuring sensor for measuring the distance from the manually driven vehicle 100 to the step S is provided, the measurement result of the distance measuring sensor is used instead of calculating the distance from the manually driven vehicle 100 to the step S. May be acquired. One example of the distance measuring sensor is an ultrasonic sensor.

  Next, in step S204, the control unit 21 calculates an entry angle of the manually driven vehicle 100 to the step S. The control unit 21 can calculate the angle of approach to the step S by calculating the angle formed by the two linear expressions described above. In the present embodiment, for the sake of convenience, the approach angle is calculated after calculating the distance, but the distance may be calculated after calculating the approach angle.

  Next, in step S205, the control unit 21 determines whether or not the distance from the manually driven vehicle 100 to the step S is smaller than the set distance, and whether the angle of entry to the step S is smaller than the set angle. As the set distance, a distance suitable for starting the driving of the support device 240 is set. The set distance is, for example, 50 cm. When the manually driven vehicle 100 enters the step S with an approach angle smaller than the set angle, an angle is set at which the traveling by the manually driven vehicle 100 may become difficult. The set angle is, for example, 10 degrees.

  If the control unit 21 determines in step S205 that the distance to the step S is equal to or greater than the set distance or that the angle of approach to the step S is equal to or greater than the set angle, the control unit 21 generates and notifies a first control signal in step S206. Output to device N. The lighting device 210 illuminates the step S in the first lighting mode. In the first lighting mode, for example, the step S is illuminated with illuminance higher than usual. The display device 230 displays the step S in the first display mode. In the first display mode, for example, the step S is highlighted.

  On the other hand, if the control unit 21 determines that the distance to the step S is smaller than the set distance in step S205 and that the angle of approach to the step S is smaller than the set angle, the control unit 21 generates a second control signal in step S207. Output to the notification device N. The lighting device 210 illuminates the step S in a second lighting mode different from the first lighting mode. In the second illumination mode, for example, the step S is illuminated in a blinking state. In addition, the display device 230 displays the step S in a second display mode different from the first display mode. In the second display mode, for example, the step S is highlighted, and a marker indicating the position of the step S and character information warning that the step S is approaching are displayed.

  The control unit 21 controls the support device 240 by outputting a control signal to the support device 240 in step S208. At this time, the control unit 21 may determine whether the step S exists on the right side or the left side with respect to the traveling direction of the manually driven vehicle 100. If the control device 2 determines that the step S exists on the right side of the traveling direction, the control device 2 outputs a control signal for controlling the traveling direction of the manually driven vehicle 100 to the right direction to the support device 240. The support device 240 drives the actuator in accordance with the control signal from the control device 2 and supports the steering of the manually driven vehicle 100 so that the traveling direction is turned rightward. By performing such control, the entry angle of the manually driven vehicle 100 with respect to the step S is increased, and the vehicle becomes easy to travel. The same applies when it is determined that the step S exists on the left side in the traveling direction, and the control device 2 assists the steering of the manually driven vehicle 100 so that the traveling direction is bent leftward.

  As described above, in the second embodiment, when the extraction device 1 extracts the step S to be extracted, the control device 2 notifies the occupant of the existence of the step S. Further, the control device 2 performs control for assisting steering as needed. With the above configuration, steering comfort can be improved.

  In the second embodiment, the extraction device 1 and the control device 2 are described as independent devices, but may be a device in which the extraction device 1 and the control device 2 are integrated. Also, not all components C including the lighting device 210, the display device 230, the audio output device 220, and the support device 240 need be connected to the control device 2, and one or more selected components C are connected. It should just be done.

  In the second embodiment, the steering is assisted by operating the steering device H of the manually driven vehicle 100. However, the steering assist may be assisted by fixing the steering device H. If the angle of entry into the step S is relatively shallow, the handlebar of the manually driven vehicle 100 may rotate in the opposite direction to the step S when the tire rides on the step S. Therefore, the control unit 21 may execute a support to suppress the rotation of the handlebar in the opposite direction by outputting a control signal for fixing the handlebar of the manually driven vehicle 100 to the support device 240.

(Third embodiment)
In the third embodiment, a communication system that transmits an extraction result by the extraction device 1 to an external server will be described.

  FIG. 8 is a block diagram illustrating a communication system according to the third embodiment. The communication system according to the third embodiment includes an extraction device 1 and a communication device 3 including a communication unit 34 that communicates with an external device. The communication device 3 further includes a position information acquisition unit 35. Since the internal configuration of the extraction device 1 is the same as that of the first embodiment, a detailed description thereof will be omitted. In the third embodiment, the extraction device 1 and the communication device 3 are described as independent devices, but may be a device in which the extraction device 1 and the control device 2 are integrated.

  The communication device 3 includes a control unit 31, a storage unit 32, an input unit 33, a communication unit 34, and a position information acquisition unit 35. The control unit 31 includes, for example, a CPU, a ROM, a RAM, and the like. The ROM provided in the control unit 31 stores a control program and the like for controlling the operation of each unit of the hardware provided in the communication device 3. The CPU in the control unit 31 executes the control programs stored in the ROM and the various programs stored in the storage unit 32, and controls the operation of each unit of the hardware, thereby realizing the communication device of the present application. Specifically, the control unit 31 transmits an extraction result input from the extraction device 1 through the input unit 33 to an external device. In the present embodiment, the external device is a server device 300 connected to a communication network NW such as the Internet. In FIG. 8, for simplicity, the number of the communication devices 3 communicating with the server device 300 is one, but the server device 300 and the plurality of communication devices 3 respectively connected to the plurality of human-powered vehicles 100 are mutually connected. You may comprise so that communication is possible.

  The control unit 31 is not limited to the above configuration. The control unit 31 may be any one or a plurality of control circuits including a single-core CPU, a multi-core CPU, an FPGA, a volatile or nonvolatile memory, and the like. In addition, the control unit 31 may include functions such as a clock for outputting date and time information, a timer for measuring an elapsed time from when a measurement start instruction is given to when a measurement end instruction is given, and a counter for counting the number.

  The storage unit 32 includes memories such as an EEPROM and an SRAM. The storage unit 32 stores a computer program executed by the control unit 31, data used by the computer program, and the like.

  The input unit 33 includes an interface for connecting the extraction device 1 via the signal line L3. The input unit 33 receives an extraction result output from the extraction device 1. The extraction result input to the input unit 33 may be image data indicating the extraction result, or may be character information or a signal indicating the extraction result. The input unit 33 outputs the input extraction result to the control unit 31. In the present embodiment, the extraction device 1 is connected via the signal line L3. However, it is configured such that data can be transmitted and received between the extraction device 1 and the communication device 3 via a wireless communication interface. Is also good. In wireless communication, a wireless transmission system conforming to the various communication standards described above can be used.

  The communication unit 34 includes a communication interface that connects to the communication network NW. The communication unit 34 transmits information to be notified to the server device 300 to the server device 300 via the communication network NW. Further, the communication unit 34 receives information transmitted from the server device 300 to the communication device 3 via the communication network NW.

  The position information obtaining unit 35 includes a receiving antenna that receives a GPS signal transmitted from a GPS satellite, an arithmetic circuit that obtains position information by measuring the current position of the manually driven vehicle 100 based on the received GPS signal, and the like. Prepare.

  FIG. 9 is a block diagram showing the internal configuration of the server device 300. The server device 300 includes a control unit 301, a storage unit 302, and a communication unit 303.

  The control unit 301 includes, for example, a CPU, a ROM, a RAM, and the like. The ROM included in the control unit 301 stores a control program and the like for controlling the operation of each unit of the hardware included in the server device 300. The CPU in the control unit 301 executes a control program stored in the ROM and various programs stored in the storage unit 302 to control the operation of each unit of the hardware.

  The control unit 301 is not limited to the configuration described above. The control unit 301 is not limited to a configuration including a CPU, a ROM, and a RAM. The control unit 301 may be, for example, one or a plurality of control circuits or arithmetic circuits including a GPU, an FPGA, a DSP, a quantum processor, a volatile or nonvolatile memory, and the like. Further, the control unit 301 may include functions such as a clock for outputting date and time information, a timer for measuring an elapsed time from when a measurement start instruction is given to when a measurement end instruction is given, and a counter for counting the number.

  The storage unit 302 includes a storage device such as a hard disk drive. Various computer programs executed by the control unit 301 are stored in the storage unit 302. Further, the storage unit 302 may store a learning model 310 learned based on data transmitted from the communication device 3 mounted on the manually driven vehicle 100. The learning model 310 is described by the definition information. The definition information of the learning model 310 includes structural information such as a neural network constituting the learning model 310, various parameters used in the learning model 310, and the like. Various parameters used in the learning model 310 include, for example, weights and biases between nodes constituting the neural network. The server device 300 learns the learning model 310 using teacher data including image data including the step S to be extracted and label data indicating that the image data includes the step S to be extracted. The server device 300 may learn the learning model 310 by using a large number of image data and label data prepared by an operator or the like of the server device 300 as teacher data. Further, as described in the fourth embodiment, the learning model 310 may be learned using image data and label data transmitted from the communication devices 3 mounted on the plurality of manually driven vehicles 100 as teacher data. Further, the teacher data may include image data that does not include the step S to be extracted, and label data indicating that the image data does not include the step S to be extracted.

  The communication unit 303 includes a communication interface that connects to the communication network NW. The communication unit 303 transmits information to be notified to the communication device 3 to the communication device 3 via the communication network NW. The communication unit 303 receives information transmitted from the communication device 3 to the server device 300 via the communication network NW.

  FIG. 10A is a flowchart illustrating a control procedure of the communication device 3 according to the third embodiment. The control unit 31 of the communication device 3 acquires an extraction result by the extraction device 1 in step S301. Next, in step S302, the control unit 31 determines whether or not the step S extracted by the extraction device 1 exists based on the extraction result obtained in step S301. When the control unit 31 determines that the step S extracted by the extraction device 1 does not exist in step S302, the processing according to this flowchart ends.

  When determining that the step S extracted by the extraction device 1 exists in step S302, the control unit 31 transmits information on the step S to the server device 300 from the communication unit 34 in step S303. In step S303, the communication unit 34 transmits the position information acquired by the position information acquisition unit 35 and the extraction result of the extraction device 1 to the external device.

  FIG. 10B is a flowchart illustrating a control procedure of the server device 300 according to the third embodiment. The server device 300 receives the information on the level difference S transmitted from the communication device 3 in step S401. In step S402, the server device 300 causes the storage unit 302 to store the information on the level difference S received in step S401. In addition, when the server device 300 receives the position information of the manually driven vehicle 100 together with the information of the step S, the server device 300 may store the received information of the step S in the storage unit 302 in association with the position information of the manually driven vehicle 100. Good.

  In step S403, the server device 300 generates map data based on the information on the step S stored in the storage unit 302 and the position information of the manually driven vehicle 100. The map data generated in step S403 is, for example, image data in which information on the step S is superimposed on the map.

  Further, in step S404, server device 300 determines whether or not a request to download map data has been received via communication network NW. If the server device 300 determines that the download request has not been received in step S404, the process according to the flowchart ends.

  When determining that the download request has been received in step S404, the server device 300 transmits the map data created in step S403 to the request source of the download request in step S405. The request source of the download request is not limited to the communication device 3 that has transmitted the information of the step S in step S303, but may be the communication device 3 mounted on another human-powered vehicle 100. It may be a portable terminal carried by a person. That is, in the third embodiment, the communication device 3 can receive, from an external device, the position information of another manually driven vehicle 100 and the extraction result of the other manually driven vehicle 100.

  As described above, in the third embodiment, when the extraction device 1 extracts the step S to be extracted, the communication device 3 transmits the information of the step S to the server device 300. Information can be managed. In addition, since the information of the step S and the position information are stored in the server device 300 and the map data including the information of the step S can be transmitted, for example, the human-powered vehicle 100 passing near the place where the step S exists Can be alerted.

(Fourth embodiment)
In the fourth embodiment, a description will be given of an information processing apparatus 4 including an arithmetic processing unit 42 that inputs a detection result regarding the surrounding environment of the manually driven vehicle 100 to a learning model and acquires an estimation result regarding the detection result from the learning model. More specifically, a mode in which the information processing device 4 updates the learning model 130 by downloading the learning model 310 from the server device 300 will be described. Since the configuration of the server device 300 is the same as that of the third embodiment, the description thereof will be omitted.

  FIG. 11 is a block diagram illustrating an internal configuration of the information processing device 4. The information processing device 4 includes an input unit 41, an arithmetic processing unit 42, a storage unit 43, an output unit 44, and a receiving unit 45. The configurations of the input unit 41, the arithmetic processing unit 42, and the storage unit 43 are the same as the configurations of the input unit 11, the arithmetic processing unit 12, and the storage unit 13 in the first embodiment, respectively, and thus description thereof is omitted. In the present embodiment, the obstacle estimated by the learning model 130 is a step S extending along the traveling direction of the manually driven vehicle 100.

  In the output unit 44, the arithmetic processing unit 42 obtains an estimation result indicating that the detection result of the surrounding environment of the manually driven vehicle 100 includes an obstacle, and the distance between the manually driven vehicle 100 and the step S is the set distance. If the angle is smaller than the predetermined angle and the angle of entry of the manually driven vehicle 100 to the step S is smaller than the set angle, the warning information is output.

  The receiving unit 45 is connected to the server device 300 via a communication network NW such as the Internet. The receiving unit 45 receives the updated learning model 310 updated by additional learning based on the detection result of the surrounding environment of the manually driven vehicle 100. When receiving the selection to update to the updated learning model, the arithmetic processing unit 42 updates the learning model 130 to the updated learning model.

  FIG. 12A is a flowchart illustrating the update procedure. In step S501, the arithmetic processing unit 42 of the information processing device 4 acquires, via the input unit 41, image data captured by the imaging device P. Next, the arithmetic processing unit 42 determines whether or not the step S has been detected in step S502. For example, the arithmetic processing unit 42 may detect the level difference S by acquiring an output of a distance measurement sensor provided in the manually driven vehicle 100. Further, the arithmetic processing unit 42 may detect the level difference S by obtaining an output of a vibration sensor provided in the manually driven vehicle 100.

  When the arithmetic processing unit 42 determines that the step S is detected in step S502, in step S503, the processing unit 42 attaches label data indicating that the step S is present to the image data acquired in step S501. The arithmetic processing unit 42 stores the image data obtained in step S501 in the storage unit 43, and adds the label data indicating that the image data is the step S to the image data obtained a set time before the time when the step S is detected. May be provided.

  If the arithmetic processing unit 42 determines in step S502 that the step S has not been detected, in step S504, the image data acquired in step S501 is provided with label data indicating that the image data is not the step S.

  Next, in step S505, the arithmetic processing unit 42 transmits the image data to which the label data has been added to the server device 300 via the communication unit 34.

  Next, in Step S506, the arithmetic processing unit 42 receives the updated learning model 310 transmitted from the server device 300 by the receiving unit 45. Next, in step S507, the arithmetic processing unit 42 determines whether the learning model 130 needs to be updated. For example, the input unit 41 may include a selection button for accepting the selection of the update necessity, and determine whether the update is necessary according to the operation of the selection button. In addition, the arithmetic processing unit 42 may inquire of the portable terminal carried by the passenger via the receiving unit 45 about the necessity of the update, and may determine whether the update is necessary based on the response from the portable terminal.

  When the arithmetic processing unit 42 determines that the update is necessary in step S507, in step S508, the arithmetic processing unit 42 updates the learning model 130 stored in the storage unit 43 with the updated learning model 310 received in step S506.

  If the arithmetic processing unit 42 determines in step S507 that updating is unnecessary, the processing according to this flowchart ends without updating the learning model 130.

  FIG. 12B is a flowchart illustrating the update procedure. In step S601, the control unit 301 of the server device 300 receives, via the communication unit 303, the image data to which the label data is transmitted from the information processing device 4.

  Next, in step S602, when receiving the image data to which the label data is added, the control unit 301 performs learning by using the image data and the label data indicating whether the image data includes the step S as teacher data. The additional learning of the model 310 is performed. By this additional learning, the structure information of the learning model 310 and various parameters used in the learning model 310 are updated, and the learning model 310 is updated. The additional learning in step S602 does not need to be performed each time image data is received, and may be performed, for example, when a predetermined number of image data is received, or at an appropriate timing such as a periodic timing. Further, the control unit 301 may additionally learn the learning model 310 using only image data to which label data indicating that the step S is given as teacher data.

  Next, in step S603, the control unit 301 reads the learning model 310 after the additional learning from the storage unit 302, and transmits the learning model 310 to the communication device 3 as an updated learning model.

  As described above, in the fourth embodiment, the server device 300 additionally learns the learning model 310 based on the image data transmitted from the information processing device 4 mounted on each of the plurality of manually driven vehicles 100. Further, the information processing device 4 mounted on the manually driven vehicle 100 can update the updated learning model transmitted from the server device 300 as necessary, and based on the updated learning model, can accurately determine the information on the traveling environment. Can be extracted well.

  In the fourth embodiment, the information processing apparatus 4 is configured to label whether or not the image data includes the step S. However, during the traveling of the manually driven vehicle 100 including the outputs of the distance measurement sensor, the vibration sensor, and the like May be transmitted to the server 300 together with the image data. That is, the teacher data includes travel information detected during travel of the manually driven vehicle 100. In this case, the server device 300 may execute additional learning of the learning model 310 based on the teacher data including the traveling information transmitted from the communication device 3.

  In the first to fourth embodiments, the step S extending along the traveling direction of the manually driven vehicle 100 is an extraction target, but the extraction target is not limited to this, and for example, a gutter, a manhole, unevenness of a road surface, An object (vehicle, person, animal, or the like) existing on a slippery road surface or around the manually driven vehicle 100 may be extracted. By using the learning model 130 learned as teacher data including the image data including the extraction target and the label data indicating that the extraction target is included, the extraction target is extracted from the newly input image data.

  The imaging direction of the imaging device P is not limited to the front of the manually driven vehicle 100. In order to extract information on an obstacle such as another vehicle existing behind, to the left or to the right of the manually driven vehicle 100, an image of the rear, left or right side of the manually driven vehicle 100 is taken using the imaging device P. It may be configured to acquire the image data obtained.

  In the first to fourth embodiments, the configuration using the image data by the imaging device P as the detection result of the surrounding environment of the manually driven vehicle 100 has been described. However, the detection result of the surrounding environment of the manually driven vehicle 100 has been described. Is not limited to image data. For example, as a detection result of the surrounding environment of the manually driven vehicle 100, distance data from the manually driven vehicle 100 to an obstacle measured using a distance measuring sensor such as an ultrasonic sensor is used alone or in combination with image data. May be used.

  The embodiments disclosed this time are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

DESCRIPTION OF SYMBOLS 1 Extraction device 11 Input part 12 Operation processing part 13 Storage part 14 Output part 130 Learning model

JP 2006-123569 A

Claims (23)

  An extraction apparatus comprising: an extraction unit configured to extract information on a traveling environment of the manually driven vehicle from a detection result of a surrounding environment of the manually driven vehicle by a learning algorithm.   The extraction device according to claim 1, wherein the learning algorithm includes a deep learning algorithm.   The extraction device according to claim 1, wherein the detection result includes an image around the manually driven vehicle.   The extraction device according to any one of claims 1 to 3, wherein the information on the traveling environment includes information on an obstacle.   The extraction device according to claim 4, wherein the information on the obstacle includes information on a step extending along a traveling direction of the manually driven vehicle. An extraction device according to any one of claims 1 to 5,
A control device for controlling components of the manually driven vehicle based on an extraction result of the extraction device.   The control system according to claim 6, wherein the component includes a notification device.   The control system according to claim 7, wherein the notification device includes a lighting device.   The control system according to any one of claims 6 to 8, wherein the component includes an assist device for a steering device of the manually driven vehicle. An extraction device according to any one of claims 1 to 5,
A communication device including a communication unit that communicates with an external device,
The communication system, wherein the communication unit transmits an extraction result of the extraction device to the external device. The communication device further includes a position information acquisition unit that acquires position information,
The communication system according to claim 10, wherein the communication unit transmits the position information acquired by the position information acquisition unit and an extraction result of the extraction device to an external device.   The communication system according to claim 11, wherein the external device generates information related to a map from the position information from the communication device and the extraction result.   The communication system according to claim 11, wherein the communication device receives, from the external device, position information of another manually driven vehicle and an extraction result of another manually driven vehicle. An input layer into which detection results about the surrounding environment of the manually driven vehicle are input,
An output layer that outputs an estimation result indicating whether the detection result includes an obstacle,
An intermediate layer based on teacher data including a detection result including an obstacle and label data indicating that the detection result includes an obstacle,
A learning model used in a process of calculating in the intermediate layer based on a detection result of the surrounding environment input to the input layer and outputting an estimation result regarding the detection result from the output layer.   The learning model according to claim 14, wherein the obstacle includes a step extending along a traveling direction of the manually driven vehicle.   The learning model according to claim 14, wherein the detection result includes an image around the manually driven vehicle.   The learning model according to any one of claims 14 to 16, wherein the teacher data includes a detection result that does not include an obstacle and label data indicating that the detection result does not include an obstacle.   The learning model according to any one of claims 14 to 17, wherein the teacher data includes traveling information detected during traveling of the manually driven vehicle.   An arithmetic processing unit that inputs a detection result about the surrounding environment of the manually driven vehicle to the learning model according to any one of claims 14 to 18, and acquires an estimation result regarding the detection result from the learning model, Information processing device. A receiving unit that receives an updated learning model updated by additional learning based on the detection result about the surrounding environment of the manually driven vehicle,
20. The information processing apparatus according to claim 19, wherein, when a selection to update to the updated learning model is received, the learning model is updated to the updated learning model. The obstacle estimated by the learning model is a step extending along the traveling direction of the manually driven vehicle,
The arithmetic processing unit acquires an estimation result indicating that the detection result includes the obstacle, a distance between the manually driven vehicle and the step is less than a set distance, and 21. The information processing apparatus according to claim 19, further comprising: an output unit that outputs warning information when an angle of entry into the step is smaller than a set angle. On the computer,
A computer program for executing a process of extracting information on a traveling environment of the manually driven vehicle from a detection result of a surrounding environment of the manually driven vehicle by a learning algorithm.   A storage medium storing the computer program according to claim 22.