WO2023160276A1 - 一种数据处理方法、装置、计算机设备以及可读存储介质 - Google Patents

一种数据处理方法、装置、计算机设备以及可读存储介质 Download PDF

Info

Publication number
WO2023160276A1
WO2023160276A1 PCT/CN2023/071183 CN2023071183W WO2023160276A1 WO 2023160276 A1 WO2023160276 A1 WO 2023160276A1 CN 2023071183 W CN2023071183 W CN 2023071183W WO 2023160276 A1 WO2023160276 A1 WO 2023160276A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
database
uploaded
data packets
cache queue
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/071183
Other languages
English (en)
French (fr)
Inventor
胡灿
张玉杰
王磊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to EP23758901.5A priority Critical patent/EP4407957B1/en
Publication of WO2023160276A1 publication Critical patent/WO2023160276A1/zh
Priority to US18/390,299 priority patent/US20240129251A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00Traffic control in data switching networks
    • H04L47/10Flow control; Congestion control
    • H04L47/31Flow control; Congestion control by tagging of packets, e.g. using discard eligibility [DE] bits
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00Traffic control in data switching networks
    • H04L47/50Queue scheduling
    • H04L47/62Queue scheduling characterised by scheduling criteria
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/12Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/56Provisioning of proxy services
    • H04L67/568Storing data temporarily at an intermediate stage, e.g. caching

Definitions

  • the present application relates to the field of computer technology, and in particular to a data processing method, device, computer equipment and readable storage medium.
  • the communication between the vehicle central control system and the outside world is limited by the network environment, memory, disk space, and processing power.
  • the communication between the vehicle central control system and the outside world is limited by the network environment, memory, disk space, and processing power.
  • the embodiments of the present application provide a data processing method, device, computer equipment, and readable storage medium, which can improve the utilization rate of network resources, and can increase the success rate of uploading data packets.
  • the embodiment of the present application provides a data processing method, which is applied to a vehicle-mounted central control device, and a database and a cache queue are set in the vehicle-mounted central control device.
  • the method includes:
  • the second data packet is deleted from the database.
  • An embodiment of the present application provides a data processing device, including:
  • the database polling module is used to poll the data messages in the database to obtain the data message set to be added, and add the data message set to be added to the cache queue;
  • the queue polling module is used to poll the buffer queue to obtain a set of data messages to be uploaded, and the wireless communication connection of the vehicle-mounted central control device will perform upload processing on the set of data messages to be uploaded;
  • a data adding module configured to add the first data message to the database in response to the first data message in the set of data messages to be uploaded meeting the continuous upload failure condition;
  • the data deletion module is configured to delete the second data message from the database in response to the second data message in the set of data messages to be uploaded is successfully uploaded to the server and the second data message exists in the database.
  • An embodiment of the present application provides a computer device, including: a processor and a memory;
  • the processor is connected to the memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the computer device executes the method provided by the embodiment of the present application.
  • An embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the embodiment of the present application provided method.
  • An embodiment of the present application provides a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium.
  • the processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided by the embodiment of the present application.
  • the computer device can poll the data packets in the database to obtain the set of data packets to be added, add the set of data packets to be added to the cache queue, and then poll the cache queue to obtain the set of data packets to be added.
  • the computer device may add the first data message to the database; in response to the second data message in the set of data messages to be uploaded If the message is successfully uploaded to the server, and the second data message exists in the database, the computer device can delete the second data message from the database.
  • the data packets to be uploaded in the data packet collection to be uploaded can be directly read from the memory, avoiding reading the data packets from the disk, thereby Improve the upload efficiency of data packets.
  • the first data packet when the data packets in the data packet set to be uploaded are uploaded, in response to the first data packet in the data packet set to be uploaded meeting the continuous upload failure condition, the first data packet can be suspended , and instead store the first data message in the database, so as to avoid repeated occupation of network resources when the network environment is not good, thereby improving the utilization rate of network resources, and when the subsequent network environment improves, through By polling the database again, there is still a chance to successfully upload the first data message, thereby increasing the success rate of uploading the data message.
  • FIG. 1 is a schematic structural diagram of a network architecture provided by an embodiment of the present application.
  • FIG. 2 is a schematic diagram of a scenario for data interaction provided by an embodiment of the present application
  • Fig. 3 is a schematic flow chart of a data processing method provided by an embodiment of the present application.
  • FIG. 4 is a schematic flow diagram of data interaction provided by an embodiment of the present application.
  • FIG. 5 is a schematic flow diagram of a data processing method provided in an embodiment of the present application.
  • FIG. 6 is a schematic flow chart of data reporting provided by an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of a data processing device provided in an embodiment of the present application.
  • Fig. 8 is a schematic structural diagram of a computer device provided by an embodiment of the present application.
  • SDK Software Development Kit
  • API Application Programming Interface
  • FIG. 1 is a schematic structural diagram of a network architecture provided by an embodiment of the present application.
  • the network architecture may include an access server 2000, a data server cluster and a terminal device cluster.
  • the terminal device cluster may specifically include one or more terminal devices, and the number of terminal devices in the terminal device cluster will not be limited here.
  • the plurality of terminal devices may specifically include terminal device 3000a, terminal device 3000b, terminal device 3000c, ..., terminal device 3000n; terminal device 3000a, terminal device 3000b, terminal device 3000c, ..., terminal device 3000n may be respectively
  • a direct or indirect network connection is made with the access server 2000 through wired or wireless communication, so that each terminal device can perform data interaction with the access server 2000 through the network connection.
  • Each terminal device in the terminal device cluster can include: smart phones, tablet computers, laptops, desktop computers, smart home appliances, wearable devices, vehicle-mounted terminals (such as vehicle-mounted central control equipment), workshop testing terminals, etc. with data processing functions smart terminal. It should be understood that each terminal device in the terminal device cluster as shown in Figure 1 can be integrated with an application client installed. Incoming server 2000 performs data exchange. Wherein, the application client may be an independent client, or an embedded sub-client integrated in a certain client, which is not limited in this application.
  • the application clients may include browsers, vehicle clients, smart home clients, entertainment clients, multimedia clients (eg, video clients), social clients, information clients, and other clients with data processing functions.
  • the vehicle-mounted terminal can be an intelligent terminal in a smart traffic scene
  • the application client on the vehicle-mounted terminal can be the above-mentioned vehicle-mounted client
  • the operating system (Operating System, OS for short) in the vehicle-mounted terminal can be a vehicle-mounted Linux OS
  • the vehicle-mounted Android (Android) OS may also be used.
  • the vehicle-mounted Linux OS is a Linux operating system running on a car
  • the vehicle-mounted Android OS is a kind of Android operating system running on a car.
  • the access server 2000 can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, and can also provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud Cloud servers for basic cloud computing services such as communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.
  • the data server cluster may include one or more data servers, and the number of data servers in the data server cluster is not limited here.
  • a plurality of data servers specifically may include data server 4000a, data server 4000b, ..., data server 4000m, data server 4000a, data server 4000b, ..., data server 4000m can respectively communicate with access server 2000 through wired or
  • the wireless communication mode is directly or indirectly connected to a network, so that each data server can perform data interaction with the access server 2000 through the network connection.
  • the data server in the data server cluster shown in Figure 1 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or provide cloud services, cloud databases, cloud computing, cloud Cloud servers for basic cloud computing services such as functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.
  • This application can provide information support for the intelligent transportation system by uploading the data message associated with the vehicle to the intelligent transportation system, so as to realize services such as transportation and automatic driving.
  • Intelligent Traffic System Intelligent Traffic System
  • Intelligent Transportation System Intelligent Transportation System
  • advanced science and technology information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory , operations research, artificial intelligence, etc.
  • advanced science and technology information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory , operations research, artificial intelligence, etc.
  • Integrated transportation system for energy for energy.
  • the terminal device described in this embodiment of the present application may be a terminal device selected from the multiple terminal devices shown in FIG. 1 .
  • the embodiment of the present application may use the vehicle-mounted terminal device 3000a shown in FIG. 1 as the terminal device described in the embodiment of the present application.
  • An application client with a data processing function may be integrated in the terminal device.
  • the terminal device can perform data interaction with the access server 2000 through the application client.
  • the terminal device may be a terminal device that needs to upload data packets.
  • the terminal device can upload the data packets to the access server 2000, so that the access server 2000 distributes different data packets to different data servers.
  • the data server also referred to as the target data server
  • the data server described in the embodiment of the present application may be a data server selected from the multiple data servers described in FIG. 1 .
  • the data server described in this embodiment of the present application may be the data server 4000a described in FIG. 1 .
  • the data server can receive the data message uploaded by the terminal device through the access server 2000 .
  • the number of application clients in the terminal device may be one or more, and one or more terminal devices may connect to one access server 2000 .
  • the terminal device can forward data packets to different data servers, without adapting the terminal device according to the requirements of the data server, and accessing a new data server without updating the terminal device, so as to meet the Requirements of different terminal devices and different application clients for data servers.
  • the access server 2000 may distribute multiple data packets corresponding to one application client to different data servers. For example, the access server 2000 may forward the data packet B11 to the data server F11, and forward the data packet B12 to the data server F12.
  • the data packet B11 and the data packet B12 here may be data packets corresponding to the same application client.
  • the access server 2000 may also distribute data packets corresponding to multiple application clients to different data servers. For example, the access server 2000 may forward the data packet B21 to the data server F21, and the access server 2000 may forward the data packet B22 to the data server F22.
  • the data packet B21 and the data packet B22 may be data packets corresponding to different application clients.
  • the access server 2000 may also distribute data packets corresponding to multiple terminal devices to different data servers. For example, the access server 2000 may forward the data packet B31 to the data server F31, and forward the data packet B32 to the data server F32.
  • the data packet B31 and the data packet B32 may be data packets corresponding to different terminal devices.
  • the embodiment of the present application provides a data reporting solution for an application client in a terminal device.
  • the data reporting scheme can provide a caching mechanism.
  • the database and the cache queue involved in the caching mechanism can store data packets. By polling the data packets in the database, a set of data packets to be added can be obtained. By polling the data packets in the cache queue, the set of data packets to be uploaded can be obtained.
  • the upload processing of the data packets in the data packet set to be uploaded can be realized, that is, the data packets are uploaded to the access server 2000, so that the access server 2000 distributes the data packets In this way, whether the data packets in the data packet set to be uploaded are successfully uploaded or the data packets in the data packet set to be uploaded fail to be uploaded, the embodiment of the present application has a complete processing flow.
  • the data uploaded by the application client may be collectively referred to as service data
  • the data obtained after packetizing the service data may be collectively referred to as data packets.
  • FIG. 2 is a schematic diagram of a data interaction scenario provided by an embodiment of the present application.
  • the server 20a may be the access server 2000 in the above-mentioned embodiment corresponding to FIG. 1
  • the terminal device 20b may be any terminal device in the terminal device cluster in the above-mentioned embodiment corresponding to FIG.
  • the server 30b can be any data server in the data server cluster of the above-mentioned embodiment corresponding to FIG.
  • the data server 4000a shown in 1 is used as the server 30a, and the data server 4000b shown in FIG. The specific process of interaction.
  • the terminal device 20b may include a database 21a and a cache queue 22a.
  • the same or different data packets may be stored in the database 21a and the cache queue 22a.
  • the embodiment of the present application does not limit the number of data packets in the database 21a and the cache queue 22a.
  • the implementation of this application does not limit the capacity of the database 21a and the cache queue 22a.
  • the data packet X1, data packet B1, and data packet B2 are included in the database 21a as an example
  • the data packet X1, data packet X2, and data packet X3 are included in the cache queue 22a as an example.
  • the capacity of the buffer queue 22a is 6 as an example for illustration.
  • the terminal device 20b can poll the database 21a and the cache queue 22a respectively.
  • the polling periods corresponding to the database 21a and the cache queue 22a may be the same or different.
  • the embodiment of the present application does not limit the duration of the polling cycles respectively corresponding to the database 21a and the cache queue 22a.
  • Should be polled can mean that the terminal device regularly sends inquiries to the database or the cache queue, asking whether the database or the cache queue needs its services, and provides the service as soon as it is available, and then continues to repeat itself.
  • the service here may mean that the terminal device reads a certain number of data packets from the database or cache queue according to a certain regulation, and then takes the read data packets as the polling result.
  • the polling result corresponding to the database may be a set of data packets to be added, and the polling result corresponding to the cache queue may be a set of data packets to be uploaded.
  • the terminal device 20b can poll the database 21a first, and then poll the cache queue 22a, and the terminal device 20b can also poll the cache queue 22a first. Perform polling, and then poll the database 21a, and the terminal device 20b may also poll the database 21a and the cache queue 22a at the same time.
  • the terminal device 20b can poll the database 21a according to the polling period corresponding to the database 21a, and poll the cache queue 22a according to the polling period corresponding to the cache queue 22a. 22a performs polling, and the polling of the two does not interfere with each other.
  • the terminal device 20b polls the database 21a first, and then polls the cache queue 22a as an example for illustration.
  • the terminal device 20b may poll the database 21a to obtain a set of data packets to be added 21b, and add the set of data packets to be added 21b to the cache queue 22a.
  • the embodiment of the present application does not limit the number of data packets in the data packet set 21b to be added.
  • the message quantity of the data message in the data message set 21b to be added can be less than or equal to the position quantity of the idle position of cache queue 22a (that is, the capacity of cache queue 22a minus the message of the data message in cache queue 22a Quantity) and a positive integer less than or equal to the number of data packets in the database 21a.
  • the number of data packets in the data packet set 21b to be added is 2 as an example for illustration, and the 2 data packets may specifically include data packets B1 and data packets B2.
  • the terminal device 20b can use the cache queue 22a added with the data message B1 and the data message B2 as the cache queue 22b, that is, the cache queue 22b can include the data message X1, the data message X2, the data message X3, the data message B1 and data message B2 are further polled from the cache queue 22b to obtain the data message set 21c to be uploaded, and the data message set 21c to be uploaded is uploaded.
  • the embodiment of the present application does not limit the number of data packets in the data packet set 21c to be uploaded.
  • the data packets in the data packet set 21c to be uploaded can be a positive integer less than or equal to the number of packets of the data packets in the cache queue 22b.
  • the number of data packets in the data packet set 21c to be uploaded is 2 as an example for illustration, and the 2 data packets may specifically include data packet X1 and data packet X2.
  • the terminal device 20b can use the cache queue 22b polling the data message X1 and the data message X2 as the cache queue 22c (not shown in the figure), that is, the cache queue 22c can include the data message X3, the data message B1 and datagram B2.
  • the terminal device 20b may add the data message X2 to the database 21a in response to determining that the data message X2 satisfies the continuous upload failure condition, so as to re-read the data message X2 in the database 21a into the cache queue in a subsequent step, so as to implement Upload processing of the data message X2; the terminal device 20b may respond to determining that the upload of the data message X1 is successful, upload the data message X1 to the server 20a, so that the server 20a distributes the data message X1 to different data servers, for example, The server 20a can distribute the data message X1 to the server 30a.
  • the terminal device 20b may determine whether the data packet X1 exists in the database 21a, and delete the data packet X1 from the database 21a in response to the existence of the data packet X1 in the database 21a.
  • the database 21a may include the data packet X2, the data packet B1 and the data packet B2.
  • the terminal device 20b can poll the database 21a and the cache queue 22c respectively again. There is a fixed time interval (also referred to as a polling cycle duration) between this polling and the previous polling. The embodiment of the present application does not limit the duration of the time interval here, for example, the time interval here may be 5 seconds. Wherein, the terminal device 20b may poll the database 21a to obtain a new set of data packets to be added, and add the new set of data packets to be added to the cache queue 22c.
  • a polling cycle duration also referred to as a polling cycle duration
  • the new set of data messages to be added can include data message X2, and the cache queue 22c (which can be called the latest cache queue (not shown in the figure)) added with data message X2 can include data message X3, data Telegram B1, data telegram B2 and data telegram X2.
  • the terminal device 20b can obtain a new set of data packets to be uploaded by polling from the latest cache queue.
  • the new set of data packets to be uploaded may include data packet X3 and data packet B1, and at this time, the new set of data packets to be uploaded may be removed from the latest cache queue.
  • the terminal device 20b in response to the successful upload of the data message X3, can upload the data message X3 to the access server 20a, so that the access server 20a distributes the data message X3 to different data servers.
  • the access server 20a may distribute the data packet X3 to the data server 30b.
  • the terminal device 20b can determine whether the data packet X3 exists in the database 21a, and in response to the absence of the data packet X3 in the database 21a, the terminal device 20b does not need to perform other operations.
  • the embodiment of the present application can provide a caching mechanism, which can use the caching queue to store the data packets to be uploaded (that is, the data packets in the data packet collection to be uploaded), and then process the data packets to be uploaded.
  • the data packets in the packet set are uploaded, different responses can be made to the success or failure of the upload.
  • the data message uploaded successfully the data message uploaded successfully can be deleted from the database when the data message uploaded successfully exists in the database;
  • the data packets that fail to be uploaded are added to the database. Therefore, the different responses made by the terminal device to the successfully uploaded data packets and the failedly uploaded data packets can improve the upload efficiency of the data packets while ensuring the upload success rate of the data packets.
  • FIG. 3 is a schematic flowchart of a data processing method provided by an embodiment of the present application.
  • the method may be executed by a server, may also be executed by a terminal device, and may also be executed jointly by the server and the terminal device.
  • the server may be the server 20a in the implementation corresponding to FIG. 2 above
  • the terminal device may be the terminal device 20b in the implementation corresponding to FIG. 2 above.
  • the method is executed by a terminal device as an example for description.
  • the terminal device may be a vehicle-mounted central control device.
  • the vehicle central control device is provided with a database and a cache queue.
  • the database can be set in the disk (for example, hard disk) of the vehicle-mounted central control device, and the cache queue can be set in the memory of the vehicle-mounted central control device.
  • the data processing method may include the following steps S101-S104.
  • Step S101 poll the data packets in the database to obtain a set of data packets to be added, and add the set of data packets to be added to a cache queue.
  • the terminal device can call the database thread to poll the data packets in the database through the reporting component, and obtain the set of data packets to be added.
  • the terminal device can send the set of data packets to be added to the reporting component through the database thread, and add the set of data packets to be added to the cache queue through the reporting component.
  • the reporting component may also include a reporting thread, and the terminal device may send the set of data packets to be added to the reporting thread through the database thread, and then add the set of data packets to be added to the cache queue through the reporting thread.
  • the reporting thread here can be used to store data packets in the cache queue and read data packets from the cache queue.
  • the database thread here can be used It is used to store data packets into the database, read data packets from the database, and so on.
  • the upload processing of data packets is realized through the inter-thread communication mechanism, which can improve the upload efficiency of data packets without affecting the operating performance of the system.
  • the terminal device can call the database thread through the reporting component.
  • the data packets are sorted to obtain the sorted data packets.
  • the terminal device may poll K data packets from the sorted data packets, and determine the K data packets as the set of data packets to be added.
  • K here may be a positive integer.
  • the first warehousing timestamp can represent the timestamp when the data message in the database is written into the database (for example, T1 time), and can also represent the modification of the original time stamp of the data message in the database (such as T1 time)
  • the time stamp after (for example, T2 time), where the T1 time may be the previous time of the T2 time, before the T2 time.
  • the time stamp after modifying the original time stamp can refer to the moment when the data message meets the continuous upload failure condition (for the continuous upload failure condition, please refer to the description of step S202-step S203 below), that is, the time stamp is obtained by meeting the continuous upload failure condition. Update the timestamp when the data packet was originally written into the database.
  • K here may be a positive integer less than or equal to the number of free positions in the cache queue.
  • K may be a positive integer less than or equal to the number of packets in the database.
  • K is a fixed value (for example, K is equal to 10).
  • the terminal device may obtain K data packets from the database by polling.
  • the terminal device can poll the database to obtain (K-1) data packets.
  • the terminal device can poll the database to obtain (K-1) data packets.
  • the terminal device can poll the database to obtain (K-1) data packets.
  • the terminal device when there is no data message in the database, that is, when the database is empty, the terminal device does not need to perform the step of polling the data message in the database, so that the set of data messages to be added cannot be obtained. , so that there is no need to perform the step of adding the set of data packets to be added to the cache queue.
  • K is equal to 0.
  • Step S102 polling the cache queue to obtain a set of data packets to be uploaded, and performing upload processing on the set of data packets to be uploaded.
  • the buffer queue may be polled to obtain a set of data packets to be uploaded, and the set of data packets to be uploaded is uploaded through the wireless communication connection of the vehicle-mounted central control device.
  • the terminal device can obtain S data packets from the cache queue according to the queue positions of the data packets in the cache queue in the reporting component, and determine the S data packets A collection of data packets to be uploaded. Wherein, S here may be a positive integer.
  • the terminal device can upload and process the set of data packets to be uploaded based on the network component (that is, the network SDK).
  • the S data packets polled by the terminal device from the cache queue can be the first S data packets in the cache queue, that is, the polled S data packets
  • the queue position of the file in the cache queue is higher.
  • the front queue position means that the data packets enter the cache queue first.
  • the data packets that enter the cache queue first can be reported first, which ensures the reporting fairness among data packets and can also reduce The situation that the data message with a long time is not reported for a long time occurs.
  • FIG. 4 is a schematic flow chart of data interaction provided by an embodiment of the present application.
  • the overall interaction process shown in Figure 4 may include a terminal system side and a server, the terminal system side may include a business layer, a reporting component, and a network component, and the server may include an access layer and multiple data backgrounds.
  • the reporting components may have different reporting strategies, and the network components may have different network knowledge.
  • the reporting component is a component used in the terminal device to upload the data of the business layer to the server.
  • the reporting component may be a component of the operating system in the terminal device.
  • the business layer includes applications, such as those installed and running on the operating system.
  • the reporting component may be a component of a software platform running in the operating system of the terminal device.
  • the business layer includes applications, such as applications running on the above-mentioned software platforms, applets (lightweight applications that can be used without installation).
  • the reporting component provides various APIs for application programs and applets to call, thereby triggering the reporting component to execute the corresponding functions of each API.
  • the application client can be used as the business layer.
  • the business layer After the business layer is connected to the reporting component, it can use the API to report the data message, and the reporting component can store the data message in the database or cache queue.
  • the network layer can read data packets from the database or cache queue, upload the data packets to the access layer of the server through the API provided by the internal network module, and the access layer can forward the data packets to different data backgrounds .
  • the access layer can be an access server
  • the data background can be a data server, that is, the terminal system can upload data packets to the access server through network components, so that the access server can distribute the data packets to different data servers, Therefore, the data message can be forwarded to any data server without modifying the application client.
  • the embodiment of the present application does not limit the number of data servers.
  • the data server includes the data background F1, the data background F2 and the data background F3 as an example for illustration.
  • the terminal device may not need to perform the step of polling the data packets in the cache queue, so that no data to be uploaded can be obtained message collection, so that there is no need to perform the step of uploading the data message to be uploaded.
  • S is equal to 0.
  • S may be a positive integer less than or equal to the number of packets in the buffer queue.
  • S is a fixed value (for example, S is equal to 5).
  • the terminal device may obtain S data packets from the cache queue by polling.
  • the terminal device can obtain all the data packets from the cache queue by polling.
  • the terminal device can select from The (S-M) data packets are obtained by polling in the cache queue, and then the (S-M) data packets are determined as a set of data packets to be uploaded. In this way, it can be ensured that the maximum number of data packets uploaded by the terminal device at the same time is S, so as to prevent the uploading process of data packets from occupying too many network resources.
  • M here may be a positive integer less than or equal to S.
  • the terminal device may accumulate the number of consecutive upload failures of the first data packet. In response to the number of consecutive upload failures reaching the number threshold, the terminal device may determine that the first data packet satisfies the continuous upload failure condition. In response to the number of consecutive upload failures not reaching the number threshold, the terminal device may determine that the first data packet does not meet the continuous upload failure condition, and re-add the first data packet to the cache queue through the reporting component.
  • the embodiment of the present application does not limit the specific value of the number of times threshold, and the embodiment of the present application takes the number of times threshold equal to 3 as an example for illustration.
  • the first data packet may be data packet B, and when uploading data packet B for the first time, the terminal device may set the number of consecutive upload failures of data packet B to 0.
  • the terminal device can accumulate the number of consecutive upload failures of the data message B to 1, and at this time, the number of consecutive upload failures is less than the threshold value, and the terminal device can re-add the data message to the cache queue. Wait for the next poll of the cache queue.
  • the terminal device can obtain the above-mentioned data message B by polling from the cache queue at the next polling time, or can obtain the above-mentioned data message B from the cache queue at the next polling time or the next next polling time , the embodiment of the present application does not limit the time for polling to obtain the data packet B.
  • the terminal device polls the data packet B again from the cache queue, it can upload the data packet B for the second time, and in response to the failure to upload the data packet B again, the terminal device can upload the The cumulative number of consecutive upload failures is 2, and so on, until the cumulative number of consecutive upload failures of data packet B is 3, and the terminal device can determine that data packet B meets the continuous upload failure condition.
  • Step S103 in response to the first data packet in the set of data packets to be uploaded meeting the continuous upload failure condition, adding the first data packet to the database.
  • step S103 "adding the first data packet to the database” refers to storing the first data packet in the database.
  • step S101 the data message in the database is polled to obtain the set of data messages to be added, each data message to be added is deleted from the database, then in step S103, the first Datagrams are written to the database.
  • step S101 the data packets in the database are polled in step S101 to obtain the set of data packets to be added
  • each data packet to be added is still retained in the database, then in step S103, the The first data message is written into the database again.
  • the terminal device may acquire the data type of the first data packet.
  • the terminal device may determine the current timestamp as the first through the reporting component.
  • the warehousing time stamp of the data message ie the second warehousing time stamp
  • the first data message and the second warehousing time stamp are sent to the database thread
  • the first data message and the second warehousing time stamp are sent by the database thread
  • the timestamp association is stored in the database.
  • the first data message is obtained by grouping multiple service data
  • the data type of the first data message is determined by the data types of the multiple service data in the first data message. It can be understood that the data types of the multiple service data in the first data packet are the same, and the terminal device can obtain multiple service data with the same data type, and then group the multiple service data with the same data type is the first data packet. In this way, the data types of the multiple service data in the first data packet are all non-discardable types, or the data types of the multiple service data in the first data packet are all discardable types.
  • the data types of multiple service data in the first data message are non-discardable types, and the data types of the first data message are non-discardable types; the data of multiple service data in the first data message The types are all discardable types, and the data type of the first data packet is a discardable type.
  • the data types of the multiple service data in the first data packet may be different, and the terminal device may obtain multiple service data of different data types, and then group multiple service data with different data types is the first data packet.
  • the data types of the multiple service data in the first data message are all non-discardable types, or the data types of the multiple service data in the first data message are all discardable types, or the data types of the multiple service data in the first data message are all discardable types.
  • the data type of a part of the business data is a non-discardable type, and the data type of another part of the business data is a discardable type.
  • the data type of the first data message is a non-discardable type;
  • the data type of the first data packet is a discardable type;
  • the data type of can be a discardable type.
  • the service data of the discardable type may represent unimportant service data uploaded by the terminal device
  • the service data of the non-discardable type may represent important service data uploaded by the terminal device.
  • the embodiment of the present application can reduce the loss of data as much as possible. By setting the non-discardable type, the reporting integrity of important data can be ensured. By setting the discardable type, the storage resources of the system can be balanced, thereby improving the upload efficiency of data messages.
  • the quantity threshold represents the maximum quantity of data packets that the database can hold.
  • the number of data packets in the database is less than the number threshold, indicating that there is free capacity in the database. If there is free capacity in the database, the terminal device may add the first data packet to the database through the reporting thread in the reporting component.
  • the embodiment of the present application does not limit the specific value of the quantity threshold. For ease of understanding, the embodiment of the present application takes the quantity threshold equal to 5000 as an example for description.
  • the terminal device in response to the data type being a discardable type, may delete the first data packet through the reporting component. Wherein, the terminal device may delete the first data packet through the reporting thread in the reporting component. At this time, neither the database nor the cache queue contains the first data packet.
  • the terminal device in response to the fact that the data type is a non-discardable type, and the number of data packets in the database is less than the quantity threshold, and there is a first data packet in the database, the terminal device can obtain the first data packet through the reporting component.
  • the update warehousing timestamp of the message send the first data message and the updated warehousing timestamp to the database thread, and update the second warehousing timestamp of the first data message in the database to the updated warehousing time stamp through the database thread timestamp.
  • updating the storage time stamp may represent the time stamp when the first data packet is rewritten into the database.
  • the terminal device may send the first data packet and the updated storage time stamp to the database thread through the reporting thread in the reporting component.
  • the terminal device in response to the fact that the data type is a non-discardable type, and the number of data packets in the database is greater than or equal to the quantity threshold, the terminal device can delete the first data packet through the reporting component, in other words, in response to the database The number of data packets in the data packet is greater than or equal to the quantity threshold, then the terminal device can delete the first data packet through the reporting thread in the reporting component.
  • Step S104 in response to the second data packet in the set of data packets to be uploaded is successfully uploaded to the server, and the second data packet exists in the database, delete the second data packet from the database.
  • the terminal device in response to the fact that the second data packet in the set of data packets to be uploaded is successfully uploaded to the server, and the second data packet does not exist in the database, the terminal device does not need to perform other steps. At this time, neither the database nor the cache queue contains the second data packet.
  • the access layer In response to the successful upload of the second data message in the set of data messages to be uploaded to the access layer (that is, the access server), the access layer can be used to unpack the second data message, and obtain The business data is distributed to the server (that is, the data server).
  • the access server may forward the service data in the second data message to the same data server, or may forward the service data in the second data message to a different data server.
  • the set of data packets to be uploaded may also include data packets other than the first data packet and the second data packet.
  • data packets other data packets by the terminal device please refer to the above description of the first data packet.
  • the description of the process of uploading the file and the second data message will not be repeated here.
  • the terminal device When the terminal device adds the first data message to the database or updates the second storage time stamp of the first data message in the database, it can poll the data message in the database in a subsequent step to obtain The above first data message, and then add the first data message obtained by polling to the cache queue, so that when the data message in the cache queue is polled in the subsequent steps, the first data message is obtained by polling, so that Realize re-uploading of the first data packet.
  • the data packets to be uploaded in the data packet collection to be uploaded can be directly read from the memory, avoiding reading the data packets from the disk, thereby Improve the upload efficiency of data packets.
  • the first data packet can be suspended
  • the first data message is stored in the database instead, so as to avoid repeated occupation of network resources when the network environment is not good, thereby improving the utilization rate of network resources.
  • FIG. 5 is a schematic flowchart of a data processing method provided by an embodiment of the present application.
  • the method may be executed by a server, may also be executed by a terminal device, and may also be executed jointly by the server and the terminal device.
  • the server may be the server 20a in the implementation corresponding to FIG. 2 above
  • the terminal device may be the terminal device 20b in the implementation corresponding to FIG. 2 above.
  • the method is executed by a terminal device as an example for description.
  • the data processing method may include the following steps S201-S204.
  • step S201 the service data uploaded by the service layer is obtained, and the service data is packaged to generate a third data packet.
  • the terminal device can obtain the business data uploaded by the business layer in the business thread, and send the business data to the reporting component through the business thread. If the data volume of the service data satisfies the grouping condition, the terminal device may perform grouping processing on the service data based on the reporting component to generate a third data packet.
  • the business reporting data will eventually be forwarded to the dedicated thread for reporting (that is, the reporting thread) through the inter-thread communication mechanism.
  • different business data can be sent by different business threads in the business layer (that is, the application client).
  • the terminal device may wait for a period of time, and then package the service data after the data volume of the acquired service data meets the grouping condition.
  • the terminal device may determine that the data volume of the service data satisfies the grouping condition. In some embodiments, if the data volume of the service data does not reach the data volume threshold, the terminal device may determine that the data volume of the service data does not satisfy the grouping condition.
  • the reporting component can provide background configuration capabilities. That is, the reporting component may receive an implementation solution configured by a server (hereinafter referred to as a configuration server) corresponding to the reporting component, and execute its function according to the configured implementation solution.
  • the server corresponding to the reporting component may refer to a server for managing, configuring and updating the reporting component.
  • the server may be a server set by the provider of the reporting component. For example, when the reporting component is a software platform running on an operating system, the corresponding server may be a cloud server provided by a provider of the software platform.
  • the server can update the implementation scheme configured in the reporting component at any time, or according to instructions sent by the user of the reporting component through other means (for example, through the reporting component management program in the smart phone).
  • the implementation plan of cloud configuration can be whether to report (that is, to determine whether to implement this plan), the size of the package (that is, the data volume of business data), the length of the cache queue (that is, the capacity of the cache queue), the capacity of the database, etc.
  • the embodiment of the present application can configure the implementation plan of the reporting component without updating the terminal device, application program and reporting component, so as to achieve the purpose of controlling the reporting process and avoiding accidents caused by the reporting itself.
  • the reporting component may receive the cache capacity configured by the configuration server, and determine whether there is an idle position in the cache queue by comparing the data volume of the data packet in the cache queue with the cache capacity. In response to the amount of data packets in the buffer queue being less than the buffer capacity, allowing data to be written into the buffer queue. Responding to the fact that the amount of data packets in the cache queue is not less than the cache capacity, refusing to write data into the cache queue.
  • the reporting component may receive the database capacity configured by the configuration server, and determine whether there is an idle position in the database by comparing the data volume of the data packet in the database with the database capacity. In response to the amount of data packets in the database being less than the capacity of the database, allowing data to be written into the database. Responding to the fact that the amount of data packets in the database is not less than the capacity of the database, refusing to write data into the database.
  • the reporting component may receive the amount of uploaded data configured by the configuration server.
  • the reporting component may use the amount of uploaded data as a limiting condition to obtain the set of data packets to be uploaded.
  • the data volume of the set of data packets to be uploaded is not greater than the uploaded data volume.
  • Packetization refers to the merging and compression of business data. If the business data uploaded to the access server is not combined and compressed each time, the network utilization rate will be greatly reduced. Especially in the case of a poor network in the vehicle environment, merging and compressing business data can not only improve network utilization, but also improve the success rate of uploading and the timeliness of reporting.
  • the service data may be a device log associated with an application client in the terminal device.
  • the device log may include log records in the vehicle-mounted terminal.
  • Log records may include abnormal problems recorded when the in-vehicle client encounters abnormal functions.
  • the log records may also include vehicle speed information and mileage information of the vehicle-mounted terminal during driving.
  • the log records may also include the driving track of the vehicle-mounted terminal during driving.
  • the log records may also include driving video data captured by the driving recorder on the vehicle terminal. The data recorded in the log record is not listed here.
  • the data server may be a server for performing data processing on log records.
  • the data server can summarize the abnormal problems encountered by the in-vehicle client during use, and provide the in-vehicle client with solutions corresponding to the abnormal problems. If the abnormal problem cannot be uploaded to the data server in a timely and accurate manner, then there is no way to obtain enough information to deal with the problem, and if the problem cannot be dealt with in a timely manner, it will lead to the loss of users.
  • the data server can analyze the vehicle speed information and mileage information of the vehicle terminal to determine whether the vehicle terminal has overspeed behavior, and then when the vehicle terminal has overspeed behavior, it will prompt the vehicle terminal with an overspeed warning to improve the safety of the vehicle terminal.
  • the data server can analyze the driving trajectory of the vehicle-mounted terminal and combine it with the current road condition information to achieve accurate and timely car-machine services, or provide automatic driving services.
  • the car-machine services here can provide avoidance The function of congested road sections.
  • the data server can monitor the running status information of the vehicle in real time through the driving video data captured by the driving recorder, so that there is no need to copy the driving video data from the vehicle terminal, so as to achieve the purpose of remote control and emergency command.
  • related data such as abnormal problems in equipment logs, vehicle speed information, mileage information, driving trajectory and driving video data, etc.
  • vehicle speed information when the above embodiments of this application are applied to specific products or technologies, user permission is required Or agree, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and national standards of the country where you are located.
  • the vehicle-mounted terminal can display a prompt message "whether to record the vehicle speed and mileage information, and send the recorded information to the server", and the vehicle-mounted terminal can use the vehicle speed and mileage information as business only after the corresponding user authorization of the vehicle-mounted terminal is passed. data, and then upload the business data.
  • the terminal device may map the log file corresponding to the service layer to the virtual address space of the memory of the service layer.
  • the terminal device may assign a service identifier (also referred to as a target service identifier) to the service data in an incremental manner, and write the target service identifier into a log file.
  • the terminal device can obtain the maximum service identifier and the minimum service identifier within a certain period of time (also referred to as a target period of time) through the virtual address space.
  • the terminal device can acquire the number of deduplication service identifiers collected by the server within the target time period.
  • the terminal device can generate a report success rate according to the maximum service identifier, the minimum service identifier, and the number of deduplication service identifiers. Wherein, the monotonically increasing target service identifier can ensure the accuracy of the generated reporting success rate.
  • the server can count the number of deduplication service identifiers received within the target time period, and then return the number of deduplication service identifiers to the terminal device, so that the terminal device can obtain the maximum service identifier and minimum service identifier within the target time period from the virtual address space.
  • the service identifier and then the terminal device can generate a report success rate according to the maximum service identifier, the minimum service identifier, and the number of deduplication service identifiers.
  • the terminal device can also send the maximum service identifier and minimum service identifier obtained from the virtual address space within the target time period to the server, so that the server can count the deduplication services received within the target time period The number of identifiers, and then the server can generate the reporting success rate according to the maximum service identifier, the minimum service identifier, and the number of deduplication service identifiers.
  • the reporting success rate may be for one data server or for multiple data servers.
  • the number of deduplication service identifiers may be counted for one data server, or may be counted for multiple data servers. In this way, when the number of deduplication service identifiers is counted for one data server, the reporting success rate is for one data server.
  • the reporting success rate is for all data servers.
  • the reporting success rate for all data servers may be equivalent to that for the access server, that is, the deduplication service identification data may be counted for the access server.
  • the data server will inevitably receive repeated target service identifiers. In this way, the data server needs to deduplicate the received target service identifiers to obtain the deduplicated target service identifiers, and then The number of deduplicated target service identifiers is used as the number of deduplicated service identifiers.
  • the target business identifier can be saved to a file (ie, a log file) mapped by mmap (memory mapping, a method for memory mapping files).
  • mmap memory mapping, a method for memory mapping files.
  • This piece of memory has the characteristics that it will be written back to the file system when the process crashes. It not only ensures that the target business ID can continue to grow after the process is restarted, but also prevents the target business ID from being too late to write to the file when the application crashes, causing the target business ID to not be monotonically continuous.
  • the terminal device can map the log file of the application client on the disk into the virtual address space of the application client in the memory. This log file can be used to store the uniquely represented target business identifier.
  • the terminal device When the process of the application client crashes, The terminal device will release the memory, and the target service identifier in the file in the memory will be written back to the disk, so as to ensure the uniqueness of the target service identifier. Similarly, when the process does not crash, the target business identifier will also be written back to the disk.
  • the specific process for the terminal device to generate the reporting success rate according to the maximum service identifier, the minimum service identifier and the number of deduplication service identifiers can refer to the following formula (1):
  • logid can represent the target business identifier
  • distinct logid can represent deduplication of logids received by the access server within the target time period
  • count(distinct logid) can represent the total number of deduplicated logids (that is, the number of deduplicated service identifiers );
  • max(logid) may represent the maximum target service ID allocated within the target time period
  • min(logid) may represent the minimum target service ID allocated within the target time period
  • ((max(logid)-min(logid)) +1) may represent the number of target business tokens generated within the target time period (ie, the cumulative number of business tokens).
  • the terminal device can generate a cumulative number of service IDs according to the maximum service ID and the minimum service ID, and then generate a reporting success rate according to a ratio between the number of deduplicated service IDs and the cumulative number of service IDs.
  • the terminal device may determine whether there is an idle position in the device cache queue, and then execute step S202 if there is an idle position in the cache queue, or execute steps S203 to S204 when there is no idle position in the cache queue.
  • Step S202 in response to an idle position in the cache queue, adding the third data packet to the cache queue through the reporting component.
  • the terminal device may add the third data packet to the cache queue through the reporting thread.
  • the business data reported by the business layer will not be uploaded to the access layer immediately, but will be stored in the cache queue first.
  • the terminal environment especially in the vehicle environment
  • system resources will be limited due to various reasons, and the network may enter an unstable state at any time, resulting in a low probability of successful data upload. Therefore, adding the packaged business data (that is, the third data message) to the cache queue can save the data on the one hand, and retry after failure to improve the success rate of data message reporting; on the other hand, it can reduce the frequency of disk access , thereby improving the upload efficiency of data packets.
  • the terminal device will add the data packet and the third data packet in the data packet set to be added to the end of the cache queue, so as to read from the cache queue in subsequent polling the third data packet.
  • the terminal device may read the third data packet when polling the cache queue for the first time in the subsequent period, or may read the third data message when polling the cache queue for the second time in the subsequent period.
  • the third data message may also be read during other subsequent polling, which is not limited in this application.
  • the terminal device may also add the third data packet to other positions in the cache queue according to the packet priority of the third data packet. For example, the header position, so that the terminal device can read the third data packet in the cache queue when it polls the cache queue for the first time.
  • the packet priority here may indicate the importance of the data packets, so as to realize the purpose of uploading the data packets with higher importance in priority.
  • Step S203 if there is no idle position in the cache queue, the current timestamp is determined as the third storage time stamp of the third data message through the reporting component, and the third data message and the third storage time stamp are sent to database thread.
  • Step S204 associate and store the third data packet and the third storage time stamp in the database through the database thread.
  • Fig. 6 is a schematic flow chart of data reporting provided by the embodiment of the present application.
  • the business layer that is, the application program
  • the business layer may execute step S61, perform initialization through step S61, and then start polling through step S62.
  • Polling can include two parts, one part is to poll the database and cache queue, and the other part is to poll and send heartbeat business data every W seconds.
  • the heartbeat business data refers to a piece of business data that is reported regularly, which can prove that the business end is still working normally, and is used to analyze the operation of the business end in the data background.
  • the data type of the heartbeat service data may be a discardable type.
  • the time interval of the first part of polling may be Y seconds.
  • Y can be a positive number, for example, 5 seconds.
  • W may represent the time interval of the second part of polling, where W may be a positive number, for example, 30 seconds.
  • the terminal device may execute step S64, and poll and send heartbeat service data every W seconds through step S64.
  • the terminal device when the terminal device is polling the database and the cache queue, it can first execute step S63, read the business data stored in the database in step S63, and then execute step S65, and pass step S65 to store in the database Polling for business data.
  • Each polling can read K pieces of business data from the database, use the K pieces of business data as a set of data packets to be added, and add the set of data packets to be added to the cache queue.
  • K here may be a positive integer.
  • the terminal device can perform step S66, poll the cache queue through step S66, traverse the cache queue to obtain S data packets, use the S data packets as a set of data packets to be uploaded, and then under the condition of network connection , performing upload processing on the data message to be uploaded through step S67, that is, uploading the data message taken out of the cache queue through the network component, and uploading the data message taken out of the cache queue to the access server.
  • S can be a positive integer.
  • S may be the number of data packets to be uploaded and processed at the same time set by the reporting component. By setting the number of data packets to be uploaded and processed at the same time, network components can be prevented from occupying too many network resources.
  • step S68 in response to the failure to upload the data message in the data message set to be uploaded, the terminal device can accumulate the number of consecutive upload failures of the data message that failed to upload, and then determine through step S70 Whether the number of consecutive upload failures exceeds the threshold.
  • the terminal device may perform step S71 to write the failed data packet into the database, read and retry in subsequent polling.
  • the terminal device may perform step S71 to re-add the failed upload data packet to the cache queue, and retry in subsequent polling.
  • the data types of data packets can be divided into two types: discardable type and non-discardable type.
  • the terminal device needs to determine the data type of the data message when writing the failed data message into the database through step S71, and then when the data type of the data message is a non-discardable type, the uploaded data message will be uploaded through step S71. Failed datagrams are written to the database.
  • the terminal device when the data type of the data packet is a discardable type, the terminal device does not need to write the data packet that fails to be uploaded into the database through step S71, but directly discards the data packet that fails to be uploaded.
  • step S72 it can determine whether the number of data packets in the database exceeds the threshold, and then write the failed data packets into the database when the number of packets does not exceed the threshold. In some embodiments, when the number of packets exceeds the threshold, the terminal device may execute step S73 to discard the data packets that failed to be uploaded.
  • the terminal device When the terminal device writes the data message that fails to upload to the database, it will update a time stamp (that is, update the storage time stamp), so that every time it polls the database, it will read the latest time stamp from the database. If the data packet is too long, add it to the cache queue and continue to retry.
  • the terminal device can poll the database through LRU (Least Recently Used, least recently used).
  • the terminal device can also poll the database through algorithms such as FIFO (First Input First Output, first-in-first-out), which is not limited in this application.
  • step S69 in response to the successful upload of the data message in the set of data messages to be uploaded, the terminal device can determine whether there is the data message successfully uploaded in the database, and then there is the data message in the database. When a successfully uploaded data message is uploaded, the successfully uploaded data message is deleted from the database. In some embodiments, when the successfully uploaded data packet does not exist in the database, the terminal device does not need to perform other operations.
  • step S74 can be executed to upload new business data through step S74 , and then perform grouping processing on the business data through step S75 .
  • the terminal device can determine whether the data volume (that is, the data size) of the business data meets the grouping condition through step S76, and then execute step S71 when the data size of the business data meets the grouping condition, and through step S71 the group The data packets obtained by the packet are added to the cache queue.
  • the terminal device can perform step S77, and in step S77 wait for the service layer to upload new service data until the received service data to be packaged When the size of the data meets the grouping conditions, the grouping is completed.
  • the terminal device When the terminal device adds the data message obtained by grouping into the cache queue through step S71, it needs to determine whether there is an idle position in the cache queue, and then when there is an idle position in the cache queue, add the data message obtained by grouping into the cache queue. cache queue. In some embodiments, when there is no idle position in the cache queue, the terminal device may execute step S72, and write the data packet obtained by grouping into the database through step S72.
  • the terminal device executes step S72, it can determine whether the number of data packets in the database exceeds the threshold value, and then when the number of packets does not exceed the threshold value, write the data packets obtained by the group package to into the database. In some embodiments, when the number of packets exceeds the threshold, the terminal device may perform step S73 to discard the data packets obtained by the group of packets.
  • the reporting thread When the buffer queue in the dedicated thread for reporting (ie, the reporting thread) is full, the reporting thread needs to directly forward the data message to the dedicated thread for the database (ie, the database thread). When the reporting thread fails to report (that is, the number of consecutive upload failures exceeds the number threshold), the reporting thread needs to indirectly forward the data packet to the database thread, so that the database thread executes step S72 to write the data packet into the database. Similarly, when reading data from the database during polling, the database thread needs to read the data message from the database, and then forward the data message to the reporting thread, so that the reporting thread executes step S71 to add the data message to the cache queue.
  • the terminal device may execute the above steps S201-S204 at any time.
  • the cache queue in the above embodiment corresponding to FIG. 3 may contain the third data packet.
  • the database in the above embodiment corresponding to FIG. 3 may include the third data packet.
  • the business data after obtaining the business data to be uploaded, the business data can be packaged, and then the third data packet obtained by the package can be added to the cache queue or database, and the cache can be subsequently processed.
  • the third data packet is obtained by polling, so as to realize the upload processing of the service data in the third data packet. It can be understood that, by grouping and processing service data, the network utilization rate can be improved, and the success rate of uploading data packets and the timeliness of reporting can be improved.
  • FIG. 7 is a schematic structural diagram of a data processing device provided by an embodiment of the present application.
  • the data processing device 1 may include: a database polling module 11 , a queue polling module 12 , a data adding module 13 , and a data deleting module 14 .
  • the data processing device 1 may further include: a count accumulation module 15 , a first comparison module 16 , a second comparison module 17 , a data generation module 18 , a first processing module 19 , and a second processing module 20 .
  • the database polling module 11 is used for polling the data message in the database, obtains the set of data messages to be added, and adds the set of data messages to be added to the cache queue;
  • the database polling module 11 includes: a database polling unit 111 and a collection adding unit 112 .
  • the database polling unit 111 is configured to call the database thread through the reporting component to poll the data packets in the database in response to the idle position in the cache queue, so as to obtain a set of data packets to be added.
  • the database polling unit 111 is configured to call the database thread through the reporting component in response to an idle position in the cache queue, and in the database thread, according to the first storage time stamp of the data message in the database, The data packets in the database are sorted to obtain the sorted data packets.
  • the database polling unit 111 is configured to poll K data packets from the sorted data packets, and determine the K data packets as the set of data packets to be added; K is a positive integer.
  • the set adding unit 112 is configured to send the set of data packets to be added to the reporting component through the database thread, and add the set of data packets to be added to the cache queue through the reporting component.
  • step S101 for the specific implementation manners of the database polling unit 111 and the set adding unit 112, reference may be made to the description of step S101 in the above-mentioned embodiment corresponding to FIG. 3 , which will not be repeated here.
  • the queue polling module 12 is configured to obtain a set of data packets to be uploaded by polling from the cache queue, and perform upload processing on the set of data packets to be uploaded.
  • the queue polling module 12 is configured to respond to the presence of data packets in the cache queue, and then in the reporting component, according to the queue position of the data packets in the cache queue, obtain S data packets from the cache queue by polling.
  • a data packet determining S data packets as a set of data packets to be uploaded. S is a positive integer.
  • the queue polling module 12 is configured to perform upload processing on a set of data packets to be uploaded based on network components.
  • the data adding module 13 is configured to add the first data packet to the database in response to the first data packet in the set of data packets to be uploaded meeting the continuous upload failure condition.
  • the data adding module 13 is configured to obtain the data type of the first data message in response to the first data message in the set of data messages to be uploaded meeting the continuous upload failure condition;
  • the data adding module 13 is configured to respond to the data type being a non-discardable type, and the number of data packets in the database is less than the quantity threshold, and there is no first data packet in the database, then by reporting The component determines the current timestamp as the second storage timestamp of the first data packet, sends the first data packet and the second storage timestamp to the database thread, and sends the first data packet and the second storage timestamp through the database thread.
  • the inbound timestamp is associated and stored in the database.
  • the data adding module 13 is further specifically configured to delete the first data packet through the reporting component in response to the data type being a discardable type.
  • the data adding module 13 is also configured to respond to the data type being a non-discardable type, and the number of data packets in the database is less than the quantity threshold, and there is a first data packet in the database, then by reporting The component obtains the updated storage time stamp of the first data message, sends the first data message and the updated storage time stamp to the database thread, and sends the second storage time stamp of the first data message in the database through the database thread Update to update the storage time stamp.
  • the data adding module 13 is further configured to delete the first data packet through the reporting component in response to the fact that the data type is non-discardable and the number of data packets in the database is greater than or equal to the quantity threshold.
  • the data deletion module 14 is configured to delete the second data message from the database in response to the second data message in the set of data messages to be uploaded is successfully uploaded to the server and the second data message exists in the database.
  • the times accumulation module 15 is configured to accumulate the number of consecutive upload failures of the first data message in response to the failure to upload the first data message in the set of data messages to be uploaded;
  • the first comparison module 16 is configured to determine that the first data packet satisfies the continuous upload failure condition in response to the number of consecutive upload failures reaching the number threshold;
  • the second comparison module 17 is configured to determine that the first data packet does not meet the continuous upload failure condition in response to the number of consecutive upload failures not reaching the number threshold, and re-add the first data packet to the cache queue through the reporting component.
  • the data generation module 18 is configured to obtain the service data uploaded by the service layer, perform grouping processing on the service data, and generate a third data packet.
  • the data generation module 18 is configured to acquire the business data uploaded by the business layer in the business thread, and send the business data to the reporting component through the business thread.
  • the data generation module 18 is configured to, in response to the data volume of the business data satisfying the grouping condition, perform grouping processing on the business data based on the reporting component, and generate a third data message.
  • the first processing module 19 is configured to add the third data packet to the buffer queue through the reporting component in response to an idle position in the buffer queue.
  • the second processing module 20 is configured to use the reporting component to determine the current timestamp as the third storage timestamp of the third data packet in response to no idle position in the cache queue, and store the third data packet The message and the third storage timestamp are sent to the database thread.
  • the second processing module 20 is configured to associate and store the third data message and the third storage time stamp in the database through the database thread.
  • the data processing apparatus 1 is further configured to map the log file corresponding to the business layer to the virtual address space of the memory of the business layer.
  • the data processing device 1 is further configured to assign a target service identifier to the service data in an incremental manner, and write the target service identifier into a log file.
  • the data processing apparatus 1 is further configured to obtain the maximum service identifier and the minimum service identifier within the target time period through the virtual address space.
  • the data processing device 1 is further configured to obtain the number of deduplication service identifiers counted by the server within the target time period.
  • the data processing device 1 is further configured to generate a reporting success rate according to the maximum service identifier, the minimum service identifier, and the number of deduplication service identifiers.
  • the data processing device 1 is further configured to respond to the successful upload of the second data message in the set of data messages to be uploaded to the access layer, and the access layer is used to unpack the second data message , and distribute the unpacked business data to the server.
  • step S101-step S104 the concrete implementation manner of database polling module 11, queue polling module 12, data adding module 13 and data deleting module 14, can refer to the description to step S101-step S104 in the embodiment corresponding to above-mentioned Fig. 3, here will not Let me repeat.
  • the specific implementation manners of the times accumulation module 15, the first comparison module 16, the second comparison module 17, the data generation module 18, the first processing module 19 and the second processing module 20 can be referred to in the above-mentioned embodiment corresponding to FIG. 5
  • the description of step S201-step S204 will not be repeated here.
  • the description of the beneficial effect of adopting the same method will not be repeated here.
  • the data processing device 1 may also include a configuration receiving module, configured to receive configuration information of the configuration server, and the configuration information includes one of the following:
  • Cache capacity used to specify the maximum amount of data that is allowed to be written in the cache queue
  • Database capacity used to specify the maximum amount of data allowed to be written in the database
  • Upload data volume which is used to specify the maximum data volume of data packets uploaded at the same time.
  • the data processing device 1 may determine whether there is an idle position in the buffer queue by comparing the data amount of the data packet in the buffer queue with the buffer capacity. In response to the amount of data packets in the buffer queue being less than the buffer capacity, allowing data to be written into the buffer queue. Responding to the fact that the amount of data packets in the cache queue is not less than the cache capacity, refusing to write data into the cache queue.
  • the data processing device 1 may determine whether there is an idle position in the database by comparing the data volume of the data packets in the database with the capacity of the database. In response to the amount of data packets in the database being less than the capacity of the database, allowing data to be written into the database. Responding to the fact that the data volume of the data message in the database is not less than the capacity of the database, refusing to write data to the database.
  • the amount of uploaded data may be used as a limiting condition to obtain the set of data packets to be uploaded.
  • the data volume of the set of data packets to be uploaded is not greater than the uploaded data volume.
  • Fig. 8 is a schematic structural diagram of a computer device provided by an embodiment of the present application.
  • the computer device 1000 may include: a processor 1001 , a network interface 1004 and a memory 1005 .
  • the above computer device 1000 may further include: a user interface 1003 and at least one communication bus 1002 .
  • the communication bus 1002 is used to realize connection and communication between these components.
  • the user interface 1003 may include a display screen (Display) and a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.
  • the network interface 1004 may include a standard wired interface and a wireless interface (such as a WI-FI interface).
  • the memory 1005 can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk memory. In some embodiments, the memory 1005 may also be at least one storage device located away from the aforementioned processor 1001 . As shown in FIG. 8 , the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.
  • the network interface 1004 can provide network communication functions; and the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control stored in the memory 1005 application to:
  • the second data packet is deleted from the database.
  • the computer device 1000 described in the embodiment of the present application can execute the description of the data processing method in the embodiment corresponding to Figure 3 and Figure 5 above, and can also execute the data processing device 1 in the embodiment corresponding to Figure 7 above description and will not be repeated here. In addition, the description of the beneficial effect of adopting the same method will not be repeated here.
  • the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores the computer program executed by the aforementioned data processing device 1, and the computer program includes
  • the program instructions when the processor executes the program instructions, can execute the description of the data processing method in the embodiment corresponding to FIG. 3 and FIG. 5 above, so details will not be repeated here.
  • the description of the beneficial effect of adopting the same method will not be repeated here.
  • the technical details not disclosed in the embodiments of the computer-readable storage medium involved in the present application please refer to the description of the method embodiments of the present application.
  • the embodiment of the present application also provides a computer program product or computer program, where the computer program product or computer program may include computer instructions, and the computer instructions may be stored in a computer-readable storage medium.
  • the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor can execute the computer instructions, so that the computer device executes the description of the data processing method in the embodiment corresponding to Figure 3 and Figure 5 above, therefore, No further details will be given here.
  • the description of the beneficial effect of adopting the same method will not be repeated here.
  • the storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM) or a random access memory (Random Access Memory, RAM), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

本申请实施例提供了一种数据处理方法、装置、计算机设备以及可读存储介质。该方法可以应用于车载中控设备。车载中控设备中设置有数据库和缓存队列。该方法包括:对数据库中的数据报文进行轮询,得到待添加数据报文集合,将待添加数据报文集合添加至缓存队列;对缓存队列进行轮询得到待上传数据报文集合,通过所述车载中控设备的无线通信连接将待上传数据报文集合进行上传;响应于待上传数据报文集合中的第一数据报文满足连续上传失败条件,则将第一数据报文存储在数据库中;响应于待上传数据报文集合中的第二数据报文成功上传至服务器,且数据库中存在第二数据报文,则从数据库中删除第二数据报文。

Description

一种数据处理方法、装置、计算机设备以及可读存储介质
本申请要求于2022年02月25日提交中国专利局、申请号为202210180680.4、发明名称为“一种数据处理方法、装置、计算机设备以及可读存储介质”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机技术领域,尤其涉及一种数据处理方法、装置、计算机设备以及可读存储介质。
发明背景
与手机、个人电脑等不同,车载中控系统与外界的通信受到网络环境、内存、磁盘空间、处理能力的限制,遇到功能异常时常无法获得足够的信息来处理问题。
发明内容
本申请实施例提供一种数据处理方法、装置、计算机设备以及可读存储介质,可以提高网络资源的利用率,且可以提高数据报文的上传成功率。
本申请实施例提供了一种数据处理方法,应用于车载中控设备,车载中控设备中设置有数据库和缓存队列,该方法包括:
对数据库中的数据报文进行轮询,得到待添加数据报文集合,将待添加数据报文集合添加至缓存队列;
对缓存队列中进行轮询得到待上传数据报文集合,通过所述车载中控设备的无线通信连接将待上传数据报文集合进行上传;
响应于待上传数据报文集合中的第一数据报文满足连续上传失败条件,则将第一数据报文添加至数据库;
响应于待上传数据报文集合中的第二数据报文成功上传至服务器,且数据库中存在第二数据报文,则从数据库中删除第二数据报文。
本申请实施例提供了一种数据处理装置,包括:
数据库轮询模块,用于对数据库中的数据报文进行轮询,得到待添加数据报文集合,将待添加数据报文集合添加至缓存队列;
队列轮询模块,用于对缓存队列中进行轮询得到待上传数据报文集合,通过所述车载中控设备的无线通信连接将对待上传数据报文集合进行上传处理;
数据添加模块,用于响应于待上传数据报文集合中的第一数据报文满足连续上传失败条件,则将第一数据报文添加至数据库;
数据删除模块,用于响应于待上传数据报文集合中的第二数据报文成功上传至服务器,且数据库中存在第二数据报文,则从数据库中删除第二数据报文。
本申请实施例提供了一种计算机设备,包括:处理器和存储器;
处理器与存储器相连,其中,存储器用于存储计算机程序,计算机程序被处理器执行时,使得该计算机设备执行本申请实施例提供的方法。
本申请实施例提供了一种计算机可读存储介质,计算机可读存储介质存储有计算机程 序,该计算机程序适于由处理器加载并执行,以使得具有该处理器的计算机设备执行本申请实施例提供的方法。
本申请实施例提供了一种计算机程序产品或计算机程序,该计算机程序产品或计算机程序包括计算机指令,该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令,处理器执行该计算机指令,使得该计算机设备执行本申请实施例提供的方法。
在本申请实施例中,计算机设备可以对数据库中的数据报文进行轮询,得到待添加数据报文集合,将待添加数据报文集合添加至缓存队列,进而从缓存队列中轮询得到待上传数据报文集合,对待上传数据报文集合进行上传处理。其中,响应于待上传数据报文集合中的第一数据报文满足连续上传失败条件,则计算机设备可以将第一数据报文添加至数据库;响应于待上传数据报文集合中的第二数据报文成功上传至服务器,且数据库中存在第二数据报文,则计算机设备可以从数据库中删除第二数据报文。由此可见,通过将待上传的数据报文存储至缓存队列,可以直接从内存中读取待上传数据报文集合中的待上传的数据报文,避免从磁盘中读取数据报文,从而提高数据报文的上传效率。此外,在对待上传数据报文集合中的数据报文进行上传处理时,响应于待上传数据报文集合中的第一数据报文满足连续上传失败条件,则可以暂停对该第一数据报文的上传,转而将该第一数据报文存储至数据库,从而避免在网络环境不好的情况下重复占用网络资源,进而可以提高网络资源的利用率,在后续网络环境好转的情况下,通过重新轮询数据库,依然可以有机会成功上传该第一数据报文,进而提高数据报文的上传成功率。
附图简要说明
为了更清楚地说明本申请实施例或相关技术中的技术方案,下面将对实施例或相关技术描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。
图1是本申请实施例提供的一种网络架构的结构示意图;
图2是本申请实施例提供的一种进行数据交互的场景示意图;
图3是本申请实施例提供的一种数据处理方法的流程示意图;
图4是本申请实施例提供的一种进行数据交互的流程示意图;
图5是本申请实施例提供的一种数据处理方法的流程示意图;
图6是本申请实施例提供的一种进行数据上报的流程示意图;
图7是本申请实施例提供的一种数据处理装置的结构示意图;
图8是本申请实施例提供的一种计算机设备的结构示意图。
实施本发明的方式
下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅仅是本申请一部分实施例,而不是全部的实施例。基于本申请中的实施例,本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例,都属于本申请保护的范围。
软件开发工具包(Software Development Kit,简称SDK)指为特定的软件包、软件框架、硬件平台、操作系统等建立应用软件时的开发工具集合。应用程序接口(Application  Programming Interface,简称API)指SDK中的一些预先定义的接口或函数,或指软件系统不同组成部分衔接的约定,用来提供应用程序与开发人员基于某软件或硬件得以访问的一组例程,而又无需访问源码,或理解内部工作机制的细节。
请参见图1,图1是本申请实施例提供的一种网络架构的结构示意图。如图1所示,该网络架构可以包括接入服务器2000、数据服务器集群和终端设备集群。其中,终端设备集群具体可以包括一个或者多个终端设备,这里将不对终端设备集群中的终端设备的数量进行限定。如图1所示,多个终端设备具体可以包括终端设备3000a、终端设备3000b、终端设备3000c、…、终端设备3000n;终端设备3000a、终端设备3000b、终端设备3000c、…、终端设备3000n可以分别与接入服务器2000通过有线或无线通信方式进行直接或间接地网络连接,以便于每个终端设备可以通过该网络连接与接入服务器2000之间进行数据交互。
终端设备集群中的每个终端设备均可以包括:智能手机、平板电脑、笔记本电脑、台式计算机、智能家电、可穿戴设备、车载终端(例如车载中控设备)、车间检测终端等具有数据处理功能的智能终端。应当理解,如图1所示的终端设备集群中的每个终端设备均可以集成安装有应用客户端,当该应用客户端运行于各终端设备中时,可以分别与上述图1所示的接入服务器2000之间进行数据交互。其中,应用客户端可以为独立的客户端,也可以为集成在某客户端中的嵌入式子客户端,本申请对此不做限定。
应用客户端具体可以包括浏览器、车载客户端、智能家居客户端、娱乐客户端、多媒体客户端(例如,视频客户端)、社交客户端以及资讯类客户端等具有数据处理功能的客户端。应当理解,车载终端可以为智慧交通场景下的智能终端,该车载终端上的应用客户端可以为上述车载客户端,该车载终端中的操作系统(Operating System,简称OS)可以为车载Linux OS、也可以为车载Android(安卓)OS。其中,车载Linux OS为一种在汽车上运行的Linux操作系统,车载Android OS为一种在汽车上运行的Android操作系统。
接入服务器2000可以是独立的物理服务器,也可以是多个物理服务器构成的服务器集群或者分布式系统,还可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、CDN、以及大数据和人工智能平台等基础云计算服务的云服务器。
数据服务器集群可以包括一个或者多个数据服务器,这里不对数据服务器集群中的数据服务器的数量进行限定。如图1所示,多个数据服务器具体可以包括数据服务器4000a、数据服务器4000b、…、数据服务器4000m,数据服务器4000a、数据服务器4000b、…、数据服务器4000m可以分别与接入服务器2000通过有线或无线通信方式进行直接或间接地网络连接,以便于每个数据服务器可以通过该网络连接与接入服务器2000之间进行数据交互。
如图1所示的数据服务器集群中的数据服务器可以是独立的物理服务器,也可以是多个物理服务器构成的服务器集群或者分布式系统,还可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、CDN、以及大数据和人工智能平台等基础云计算服务的云服务器。
本申请通过将与车辆相关联的数据报文上传到智能交通系统中,可以为智能交通系统提供信息支持,以实现交通运输、自动驾驶等服务。其中,智能交通系统(Intelligent Traffic System,ITS)又称智能运输系统(Intelligent Transportation System),是将先进的科学技术(信 息技术、计算机技术、数据通信技术、传感器技术、电子控制技术、自动控制理论、运筹学、人工智能等)有效地综合运用于交通运输、服务控制和车辆制造,加强车辆、道路、使用者三者之间的联系,从而形成一种保障安全、提高效率、改善环境、节约能源的综合运输系统。
本申请实施例中所描述的终端设备(也称为目标终端设备)可以是从图1所示的多个终端设备中选择的一个终端设备。例如,本申请实施例可以将图1所示的车载终端设备3000a作为本申请实施例中描述的终端设备。该终端设备中可以集成有具备数据处理功能的应用客户端。此时,该终端设备可以通过该应用客户端与接入服务器2000之间进行数据交互。
终端设备可以为需要上传数据报文的终端设备。终端设备可以将数据报文上传至接入服务器2000,以使接入服务器2000将不同的数据报文分发至不同的数据服务器。为便于理解,本申请实施例中所描述的数据服务器(也称为目标数据服务器)可以是从图1所述的多个数据服务器中选择的一个数据服务器。例如,本申请实施例所描述的数据服务器可以是图1所述的数据服务器4000a。该数据服务器可以通过接入服务器2000接收终端设备上传的数据报文。
终端设备中的应用客户端的数量可以为一个或多个,且一个或多个终端设备可以对接一个接入服务器2000。通过接入服务器,终端设备可以将数据报文转发到不同的数据服务器,不需要按照数据服务器的要求对终端设备进行适配,在避免更新终端设备的前提下接入新的数据服务器,从而满足不同终端设备和不同应用客户端对于数据服务器的要求。
接入服务器2000可以将一个应用客户端所对应的多个数据报文分发至不同的数据服务器。比如,接入服务器2000可以将数据报文B11转发至数据服务器F11,将数据报文B12转发至数据服务器F12。这里的数据报文B11和数据报文B12可以为同一个应用客户端所对应的数据报文。一些实施例中,接入服务器2000也可以将多个应用客户端所分别对应的数据报文分发至不同的数据服务器。比如,接入服务器2000可以将数据报文B21转发至数据服务器F21,接入服务器2000可以将数据报文B22转发至数据服务器F22。数据报文B21和数据报文B22可以为不同应用客户端对应的数据报文。一些实施例中,接入服务器2000还可以将多个终端设备对应的数据报文分别分发至不同的数据服务器。比如,接入服务器2000可以将数据报文B31转发至数据服务器F31,将数据报文B32转发至数据服务器F32。数据报文B31和数据报文B32可以为不同终端设备对应的数据报文。
本申请实施例为终端设备中的应用客户端提供一种数据上报方案。该数据上报方案可以提供一种缓存机制,该缓存机制所涉及的数据库和缓存队列中可以存储有数据报文,通过对数据库中的数据报文进行轮询,可以得到待添加数据报文集合。通过对缓存队列中的数据报文进行轮询,可以得到待上传数据报文集合。因此,通过数据库和缓存队列的联合作用,可以实现对待上传数据报文集合中的数据报文的上传处理,即将数据报文上传至接入服务器2000,以使接入服务器2000将数据报文分发至数据服务器,这样,无论待上传数据报文集合中的数据报文上传成功,还是待上传数据报文集合中的数据报文上传失败,本申请实施例都具备完整的处理流程。
为便于理解,本申请实施例可以将应用客户端上传的数据统称为业务数据,本申请实施例可以将对业务数据进行组包处理后所得到的数据统称为数据报文。
图2是本申请实施例提供的一种进行数据交互的场景示意图。如图2所示,服务器20a 可以为上述图1所对应实施例中的接入服务器2000,终端设备20b可以为上述图1所对应实施例的终端设备集群中的任意一个终端设备,服务器30a和服务器30b分别可以为上述图1所对应实施例的数据服务器集群中的任意一个数据服务器,为便于理解,本申请实施例以上述图1所示的终端设备3000a作为该终端设备20b、以上述图1所示的数据服务器4000a作为该服务器30a、以上述图1所示的数据服务器4000b作为该服务器30b为例,以阐述图2所示的服务器20a、终端设备20b、服务器30a和服务器30b进行数据交互的具体过程。
如图2所示,终端设备20b中可以包括数据库21a和缓存队列22a。数据库21a和缓存队列22a中可以存储有相同或不同的数据报文。本申请实施例不对数据库21a和缓存队列22a中的数据报文的报文数量进行限定。本申请实施不对数据库21a和缓存队列22a的容量进行限定。为便于理解,这里以数据库21a中包含数据报文X1、数据报文B1和数据报文B2为例,以缓存队列22a中包含数据报文X1、数据报文X2和数据报文X3为例进行说明,这里以缓存队列22a的容量为6个为例进行说明。
如图2所示,终端设备20b可以对数据库21a和缓存队列22a分别进行轮询。数据库21a和缓存队列22a所分别对应的轮询周期可以是相同的,也可以是不同的。本申请实施例不对数据库21a和缓存队列22a所分别对应的轮询周期的周期时长进行限定。应轮询可以表示终端设备定时对数据库或者缓存队列发出询问,询问数据库或者缓存队列是否需要其服务,有即给予服务,接着不断周而复始。这里的服务可以表示终端设备从数据库或缓存队列中按照某种规定读取一定数量的数据报文,进而将读取到的数据报文作为轮询的结果。数据库所对应的轮询的结果可以为待添加数据报文集合,缓存队列所对应的轮询的结果可以为待上传数据报文集合。
其中,在数据库21a和缓存队列22a所分别对应的轮询周期相同时,终端设备20b可以先对数据库21a进行轮询、再对缓存队列22a进行轮询,终端设备20b也可以先对缓存队列22a进行轮询、再对数据库21a进行轮询,终端设备20b还可以同时对数据库21a和缓存队列22a进行轮询。其中,在数据库21a和缓存队列22a所分别对应的轮询周期不同时,终端设备20b可以按照数据库21a对应的轮询周期对数据库21a进行轮询,按照缓存队列22a对应的轮询周期对缓存队列22a进行轮询,两者的轮询互不干扰。本申请实施例以终端设备20b先对数据库21a进行轮询、再对缓存队列22a进行轮询为例进行说明。
如图2所示,终端设备20b可以从数据库21a中轮询得到待添加数据报文集合21b,将待添加数据报文集合21b添加至缓存队列22a。其中,本申请实施例不对待添加数据报文集合21b中的数据报文的报文数量进行限定。待添加数据报文集合21b中的数据报文的报文数量可以为小于或等于缓存队列22a的空闲位置的位置数量(即缓存队列22a的容量减去缓存队列22a中的数据报文的报文数量)、且小于或等于数据库21a中的数据报文的报文数量的正整数。这里以待添加数据报文集合21b中的数据报文的报文数量为2个为例进行说明,2个数据报文具体可以包含数据报文B1和数据报文B2。
终端设备20b可以将添加有数据报文B1和数据报文B2的缓存队列22a作为缓存队列22b,即缓存队列22b中可以包含数据报文X1、数据报文X2、数据报文X3、数据报文B1和数据报文B2,进而从缓存队列22b中轮询得到待上传数据报文集合21c,对待上传数据报文集合21c进行上传处理。其中,本申请实施例不对待上传数据报文集合21c中的数据报文的报文数量进行限定。待上传数据报文集合21c中的数据报文可以为小于或等于缓存 队列22b中的数据报文的报文数量的正整数。这里以待上传数据报文集合21c中的数据报文的报文数量为2个为例进行说明,2个数据报文具体可以包含数据报文X1和数据报文X2。
终端设备20b可以将轮询出数据报文X1和数据报文X2的缓存队列22b作为缓存队列22c(未在图上示出),即缓存队列22c中可以包含数据报文X3、数据报文B1和数据报文B2。
如图2所示,在待上传数据报文集合21c中的数据报文X1和数据报文X2上传处理的过程中,假设数据报文X2满足连续上传失败条件,数据报文X1上传成功,终端设备20b可以响应于确定数据报文X2满足连续上传失败条件,将数据报文X2添加至数据库21a,以在后续步骤中重新读取数据库21a中的数据报文X2到缓存队列中,以实现对数据报文X2的上传处理;终端设备20b可以响应于确定数据报文X1上传成功,将数据报文X1上传至服务器20a,以使服务器20a将数据报文X1分发至不同的数据服务器,例如,服务器20a可以将数据报文X1分发至服务器30a。此外,终端设备20b可以确定数据库21a中是否存在数据报文X1,响应于数据库21a中存在数据报文X1,则从数据库21a中删除数据报文X1。此时,数据库21a中可以包含数据报文X2、数据报文B1和数据报文B2。
终端设备20b可以对数据库21a和缓存队列22c分别进行又一次轮询。这一次轮询和上一次轮询之间相差固定的时间间隔(也可以称之为轮询周期时长)。本申请实施例不对这里的时间间隔的时长进行限定,例如,这里的时间间隔可以为5秒。其中,终端设备20b可以从数据库21a中轮询得到新的待添加数据报文集合,将新的待添加数据报文集合添加至缓存队列22c。新的待添加数据报文集合可以包含数据报文X2,添加有数据报文X2的缓存队列22c(可以称为最新缓存队列(未在图中示出))中可以包含数据报文X3、数据报文B1、数据报文B2和数据报文X2。终端设备20b可以从最新缓存队列中轮询得到新的待上传数据报文集合。其中,新的待上传数据报文集合可以包含数据报文X3和数据报文B1,此时新的待上传数据报文集合可以移出最新缓存队列。
这样,在数据报文X3上传处理的过程中,响应于数据报文X3上传成功,终端设备20b可以将数据报文X3上传至接入服务器20a,以使接入服务器20a将数据报文X3分发至不同的数据服务器。例如,接入服务器20a可以将数据报文X3分发至数据服务器30b。此外,终端设备20b可以确定数据库21a中是否存在数据报文X3,响应于数据库21a中不存在数据报文X3,则终端设备20b无需执行其他操作。
由此可见,本申请实施例可以提供一种缓存机制,该缓存机制可以使用缓存队列来存储待上传的数据报文(即待上传数据报文集合中的数据报文),进而在对待上传数据报文集合中的数据报文进行上传处理时,可以针对上传成功或上传失败的情况作出不同的响应。其中,针对上传成功的数据报文而言,可以在数据库中存在该上传成功的数据报文时,从数据库中删除该上传成功的数据报文;针对上传失败的数据报文而言,可以将该上传失败的数据报文添加至数据库。因此,终端设备针对上传成功的数据报文和上传失败的数据报文所作出的不同响应,可以在保证数据报文的上传成功率的同时,提高数据报文的上传效率。
图3是本申请实施例提供的一种数据处理方法的流程示意图。该方法可以由服务器执行,也可以由终端设备执行,还可以由服务器和终端设备共同执行。该服务器可以为上述图2所对应实施中的服务器20a,该终端设备可以为上述图2所对应实施中的终端设备20b。 本申请实施例以该方法由终端设备执行为例进行说明。本实施例中,终端设备可以为车载中控设备。车载中控设备中设置有数据库和缓存队列。其中,数据库可以设置在车载中控设备的磁盘(例如硬盘)中,缓存队列可以设置在车载中控设备的内存中。其中,该数据处理方法可以包括以下步骤S101-步骤S104。
步骤S101,对数据库中的数据报文进行轮询,得到待添加数据报文集合,将待添加数据报文集合添加至缓存队列。
若缓存队列中存在空闲位置,则终端设备可以通过上报组件调用数据库线程对数据库中的数据报文进行轮询,得到待添加数据报文集合。终端设备可以通过数据库线程将待添加数据报文集合发送至上报组件,通过上报组件将待添加数据报文集合添加至缓存队列。
上报组件(即上报SDK)中还可以包括上报线程,终端设备可以通过数据库线程将待添加数据报文集合发送至上报线程,进而通过上报线程将待添加数据报文集合添加至缓存队列。应当理解,上报线程和数据库线程均为上报组件的组成部分,这里的上报线程可以用于将数据报文存储至缓存队列、以及从缓存队列中读取数据报文等,这里的数据库线程可以用于将数据报文存储至数据库、以及从数据库中读取数据报文等。通过线程间通信机制来实现数据报文的上传处理,可以实现在不影响系统运行性能的情况下,提升数据报文的上传效率。
若缓存队列中存在空闲位置,则终端设备可以通过上报组件调用数据库线程,在数据库线程中,根据数据库中的数据报文的入库时间戳(即第一入库时间戳),对数据库中的数据报文进行排序,得到排序后的数据报文。终端设备可以从排序后的数据报文中轮询得到K个数据报文,将K个数据报文确定为待添加数据报文集合。其中,这里的K可以为正整数。
第一入库时间戳可以表示数据库中的数据报文被写入数据库时的时间戳(例如,T1时刻),也可以表示对数据库中的数据报文的原本时间戳(如T1时刻)进行修改后的时间戳(例如,T2时刻),这里的T1时刻可以为T2时刻的上一时刻,在T2时刻之前。对原本时间戳进行修改后的时间戳可以是指数据报文满足连续上传失败条件(连续上传失败条件可以参见下面步骤S202-步骤S203的说明)的时刻,即通过满足连续上传失败条件的时刻来更新该数据报文原先被写入数据库的时间戳。
这里的K可以为小于或等于缓存队列的空闲位置的位置数量的正整数。K可以为小于或等于数据库中的数据报文的报文数量的正整数。在一般情况下,K是一个固定值(例如,K等于10)。在缓存队列的空闲位置的位置数量大于K、且数据库中的数据报文的报文数量大于K时,终端设备可以从数据库中轮询得到K个数据报文。在缓存队列的空闲位置的位置数量等于(K-1)、且数据库中的数据报文的报文数量大于K时,终端设备可以从数据库中轮询得到(K-1)个数据报文。在缓存队列的空闲位置的位置数量大于K、且数据库中的数据报文的报文数量等于(K-1)时,终端设备可以从数据库中轮询得到(K-1)个数据报文。
一些实施例中,在数据库中不存在数据报文时,即数据库为空时,终端设备可以无需执行对数据库中的数据报文进行轮询的步骤,这样就获取不到待添加数据报文集合,进而就无需执行将待添加数据报文集合添加至缓存队列的步骤。此时,K等于0。
步骤S102,从缓存队列中轮询得到待上传数据报文集合,对待上传数据报文集合进行上传处理。
步骤S102中,可以对所述缓存队列进行轮询得到待上传数据报文集合,将待上传数 据报文集合通过车载中控设备的无线通信连接进行上传。
若缓存队列中存在数据报文,则终端设备可以在上报组件中,根据缓存队列中的数据报文的队列位置,从缓存队列中轮询得到S个数据报文,将S个数据报文确定为待上传数据报文集合。其中,这里的S可以为正整数。终端设备可以基于网络组件(即网络SDK)对待上传数据报文集合进行上传处理。
根据队列(例如,缓存队列)先进先出的原则,终端设备从缓存队列中轮询得到的S个数据报文可以为缓存队列中前S个数据报文,即轮询出的S个数据报文在缓存队列中的队列位置更靠前。其中,靠前的队列位置代表数据报文是先进入缓存队列的,这时就可以优先对先进入缓存队列的数据报文进行上报,保证了数据报文之间的上报公平性,也可以减少时间久远的数据报文迟迟未被上报的情况发生。
图4是本申请实施例提供的一种进行数据交互的流程示意图。如图4所示的总体交互流程可以包含终端系统端和服务器,终端系统端可以包含业务层、上报组件和网络组件,服务器可以包含接入层和多个数据后台。其中,上报组件可以具有不同的上报策略,网络组件可以具有不同的网络知识。
上报组件为终端设备中用于将业务层的数据上传到服务器的部件。
一些实施例中,上报组件可以是终端设备中的操作系统的组成部分。业务层包括应用程序,例如安装并运行在操作系统中的应用程序。
一些实施例中,上报组件可以是终端设备的操作系统中运行的一个软件平台的组成部分。业务层包括应用程序,例如在上述软件平台中运行的应用程序、小程序(无需安装即可使用的轻量级应用)。上报组件提供各种API供应用程序和小程序调用,从而触发上报组件执行各API相应的功能。
如图4所示,应用客户端可以作为业务层,业务层在接入上报组件之后,可以使用其中的API上报数据报文,上报组件可以将数据报文存储至数据库或缓存队列中。网络层可以从数据库或缓存队列中读取数据报文,通过内部的网络模块所提供的API将数据报文上传到服务端的接入层,接入层可以将数据报文转发到不同的数据后台。
接入层可以为接入服务器,数据后台可以为数据服务器,即终端系统端可以通过网络组件将数据报文上传至接入服务器,以使接入服务器将数据报文分发至不同的数据服务器,从而可以在无需修改应用客户端的情况下,将数据报文转发到任意数据服务器。本申请实施例不对数据服务器的数量进行限定,这里以数据服务器包含数据后台F1、数据后台F2和数据后台F3为例进行说明。
一些实施例中,在缓存队列中不存在数据报文时,即缓存队列为空时,终端设备可以无需执行对缓存队列中的数据报文进行轮询的步骤,这样就获取不到待上传数据报文集合,进而就无需执行对待上传数据报文进行上传处理的步骤。此时,S等于0。
这里的S可以为小于或等于缓存队列中的数据报文的报文数量的正整数。在一般情况下,S是一个固定值(例如,S等于5)。在缓存队列中的数据报文的报文数量大于S时,终端设备可以从缓存队列中轮询得到S个数据报文。在缓存队列中的数据报文的报文数量小于S时,终端设备可以从缓存队列中轮询得到全部的数据报文。
同理,一些实施例中,若在终端设备对缓存队列进行轮询时,终端设备正在对M个数据报文进行上传处理,则终端设备可以根据缓存队列中的数据报文的队列位置,从缓存队列中轮询得到(S-M)个数据报文,进而将(S-M)个数据报文确定为待上传数据报文集合。 这样可以确保终端设备在同一时间上传的数据报文的最大报文数量为S个,从而避免数据报文的上传过程占用过多的网络资源。其中,这里的M可以为小于或等于S的正整数。
响应于待上传数据报文集合中的第一数据报文上传失败,则终端设备可以累计第一数据报文的连续上传失败次数。响应于连续上传失败次数达到次数阈值,则终端设备可以确定第一数据报文满足连续上传失败条件。响应于连续上传失败次数未达到次数阈值,则终端设备可以确定第一数据报文不满足连续上传失败条件,通过上报组件将第一数据报文重新添加至缓存队列。本申请实施例不对次数阈值的具体取值进行限定,本申请实施例以次数阈值等于3为例进行说明。
比如,第一数据报文可以为数据报文B,当第一次对数据报文B进行上传处理时,终端设备可以将数据报文B的连续上传失败次数设置为0。响应于数据报文B上传失败,则终端设备可以将数据报文B的连续上传失败次数累计为1,此时连续上传失败次数小于次数阈值,终端设备可以重新将数据报文添加至缓存队列,等待下次对缓存队列进行轮询。其中,终端设备可以在下一次轮询时从缓存队列中轮询得到上述数据报文B,也可以在下下次轮询或下下下次轮询时从缓存队列中轮询得到上述数据报文B,本申请实施例不对轮询得到数据报文B的时间进行限定。当终端设备再次从缓存队列中轮询得到数据报文B时,可以第二次对数据报文B进行上传处理,响应于数据报文B再次上传失败,则终端设备可以将数据报文B的连续上传失败次数累计为2,以此类推,直到数据报文B的连续上传失败次数累计为3时,终端设备可以确定数据报文B满足连续上传失败条件。
步骤S103,响应于待上传数据报文集合中的第一数据报文满足连续上传失败条件,则将第一数据报文添加至数据库。
步骤S103中,“将第一数据报文添加至数据库”是指将第一数据报文存储在数据库中。当一些实施例中,在步骤S101中对数据库中的数据报文进行轮询得到待添加数据报文集合时,将各待添加数据报文从数据库中删除,则在步骤S103中,将第一数据报文写入数据库中。另一些实施例中,在步骤S101中对数据库中的数据报文进行轮询得到待添加数据报文集合时,各待添加数据报文仍然保留在数据库中,则在步骤S103中,可以不将第一数据报文再次写入数据库中。
具体的,响应于待上传数据报文集合中的第一数据报文满足连续上传失败条件,则终端设备可以获取第一数据报文的数据类型。响应于数据类型为不可丢弃类型,且数据库中的数据报文的报文数量小于数量阈值,且数据库中不存在第一数据报文,则终端设备可以通过上报组件将当前时间戳确定为第一数据报文的入库时间戳(即第二入库时间戳),将第一数据报文和第二入库时间戳发送至数据库线程,通过数据库线程将第一数据报文和第二入库时间戳关联存储至数据库。
其中,第一数据报文是对多个业务数据进行组包处理后所得到的,第一数据报文的数据类型是由第一数据报文中的多个业务数据的数据类型所确定的。可以理解的是,第一数据报文中的多个业务数据的数据类型是相同的,终端设备可以获取具有相同数据类型的多个业务数据,进而将具有相同数据类型的多个业务数据组包为第一数据报文。这样,第一数据报文中的多个业务数据的数据类型均为不可丢弃类型,或者第一数据报文中的多个业务数据的数据类型均为可丢弃类型。因此,在第一数据报文中的多个业务数据的数据类型均为不可丢弃类型,第一数据报文的数据类型为不可丢弃类型;在第一数据报文中的多个业务数据的数据类型均为可丢弃类型,第一数据报文的数据类型为可丢弃类型。
一些实施例中,第一数据报文中的多个业务数据的数据类型可以是不同的,终端设备可以获取不同数据类型的多个业务数据,进而将具有不同数据类型的多个业务数据组包为第一数据报文。这样,第一数据报文中的多个业务数据的数据类型均为不可丢弃类型,或者第一数据报文中的多个业务数据的数据类型均为可丢弃类型,或者第一数据报文中的一部分业务数据的数据类型为不可丢弃类型、另一部分业务数据的数据类型为可丢弃类型。因此,在第一数据报文中的多个业务数据的数据类型均为不可丢弃类型时,第一数据报文的数据类型为不可丢弃类型;在第一数据报文中的多个业务数据的数据类型均为可丢弃类型时,第一数据报文的数据类型为可丢弃类型;在第一数据报文同时包含不可丢弃类型的业务数据和可丢弃类型的业务数据时,第一数据报文的数据类型可以为可丢弃类型。
其中,可丢弃类型的业务数据可以表示终端设备所上传的不重要的业务数据,不可丢弃类型的业务数据可以表示终端设备所上传的重要的业务数据。本申请实施例可以尽可能地减少数据的丢失,通过设置不可丢弃类型,可以保证重要数据的上报完整性,通过设置可丢弃类型,可以平衡系统的存储资源,从而提高数据报文的上传效率。
其中,数量阈值表示数据库所能容纳的数据报文的最大数量。在数据库中的数据报文的报文数量小于数量阈值,表示数据库中具有空闲容量。若数据库中具有空闲容量,则终端设备可以通过上报组件中的上报线程,将第一数据报文添加至数据库。应当理解,本申请实施例不对数量阈值的具体取值进行限定,为便于理解,本申请实施例以数量阈值等于5000为例进行说明。
一些实施例中,响应于数据类型为可丢弃类型,则终端设备可以通过上报组件删除第一数据报文。其中,终端设备可以通过上报组件中的上报线程删除第一数据报文,此时,数据库和缓存队列中均不包含第一数据报文。
一些实施例中,响应于数据类型为不可丢弃类型,且数据库中的数据报文的报文数量小于数量阈值,且数据库中存在第一数据报文,则终端设备可以通过上报组件获取第一数据报文的更新入库时间戳,将第一数据报文和更新入库时间戳发送至数据库线程,通过数据库线程将第一数据报文在数据库中的第二入库时间戳更新为更新入库时间戳。其中,更新入库时间戳可以表示第一数据报文重新写入数据库时的时间戳。其中,终端设备可以通过上报组件中的上报线程将第一数据报文和更新入库时间戳发送至数据库线程。
一些实施例中,响应于数据类型为不可丢弃类型,且数据库中的数据报文的报文数量大于或等于数量阈值,则终端设备可以通过上报组件删除第一数据报文,换言之,响应于数据库中的数据报文的报文数量大于或等于数量阈值,则终端设备可以通过上报组件中的上报线程删除第一数据报文。
步骤S104,响应于待上传数据报文集合中的第二数据报文成功上传至服务器,且数据库中存在第二数据报文,则从数据库中删除第二数据报文。
一些实施例中,响应于待上传数据报文集合中的第二数据报文成功上传至服务器,且数据库中不存在第二数据报文,则终端设备无需执行其他步骤。此时,数据库和缓存队列中均不包含第二数据报文。
响应于待上传数据报文集合中的第二数据报文成功上传至接入层(即接入服务器),则接入层可以用于对第二数据报文进行解包处理,将解包得到的业务数据分发至服务器(即数据服务器)。其中,接入服务器可以将第二数据报文中的业务数据转发至相同的数据服务器,也可以将第二数据报文中的业务数据转发至不同的数据服务器。
待上传数据报文集合中还可以包括除第一数据报文和第二数据报文之外的其他数据报文,终端设备对其他数据报文进行上传处理的过程可以参见上述对第一数据报文和第二数据报文的上传处理的过程的描述,这里将不再进行赘述。
终端设备在将第一数据报文添加至数据库或者更新第一数据报文在数据库中的第二入库时间戳时,可以在后续步骤对数据库中的数据报文进行轮询时,轮询得到上述第一数据报文,进而将轮询得到的第一数据报文添加至缓存队列,以在后续步骤对缓存队列中的数据报文进行轮询时,轮询得到第一数据报文,从而实现对第一数据报文的重新上传。
由此可见,通过将待上传的数据报文存储至缓存队列,可以直接从内存中读取待上传数据报文集合中的待上传的数据报文,避免从磁盘中读取数据报文,从而提高数据报文的上传效率。此外,在对待上传数据报文集合中的数据报文进行上传处理时,响应于待上传数据报文集合中的第一数据报文满足连续上传失败条件,则可以暂停对该第一数据报文的上传,转而将该第一数据报文存储至数据库,从而避免在网络环境不好的情况下重复占用网络资源,进而可以提高网络资源的利用率。在后续网络环境好转的情况下,通过重新轮询数据库,依然可以有机会成功上传该第一数据报文,进而提高数据报文的上传成功率。
图5是本申请实施例提供的一种数据处理方法的流程示意图。该方法可以由服务器执行,也可以由终端设备执行,还可以由服务器和终端设备共同执行。该服务器可以为上述图2所对应实施中的服务器20a,该终端设备可以为上述图2所对应实施中的终端设备20b。本申请实施例以该方法由终端设备执行为例进行说明。其中,该数据处理方法可以包括以下步骤S201-步骤S204。
步骤S201,获取业务层上传的业务数据,对所述业务数据进行组包处理,生成第三数据报文。
终端设备可以在业务线程中,获取业务层上传的业务数据,通过业务线程将业务数据发送至上报组件。若业务数据的数据量满足组包条件,则终端设备可以基于上报组件对业务数据进行组包处理,生成第三数据报文。
为了不影响系统运行性能,业务上报数据最终都会通过线程间通信机制转发到上报专用线程(即上报线程),换言之,不同的业务数据可以由业务层(即应用客户端)中的不同的业务线程发送至上报组件中的上报线程,进而在上报线程中,若业务数据的数据量满足组包条件,则对业务数据进行组包处理。一些实施例中,若业务数据的数据量不满足组包条件,则终端设备可以等待一段时间,在获取到的业务数据的数据量满足组包条件之后,再对业务数据进行组包处理。
若业务数据的数据量达到数据量阈值,则终端设备可以确定业务数据的数据量满足组包条件。一些实施例中,若业务数据的数据量未达到数据量阈值,则终端设备可以确定业务数据的数据量不满足组包条件。
上报组件可以提供后台配置能力。也即,上报组件可以接收上报组件对应的服务器(以下称为配置服务器)配置的实施方案,并根据配置的实施方案执行其功能。上报组件对应的服务器可以指用于对上报组件进行管理和配置、更新的服务器。该服务器可以是由上报组件的提供者设置的服务器。例如,当上报组件为操作系统中运行的软件平台时,对应的服务器可以是该软件平台的提供者提供的云服务器。该服务器可以在任意时间,或根据上报组件的用户通过其它途径(例如通过智能手机中的上报组件管理程序)发送的指示,对上报组件中配置的实施方案进行更新。例如,云端配置的实施方案可以为是否上报(即确 定是否执行本方案)、组包的大小(即业务数据的数据量),缓存队列长度(即缓存队列的容量),数据库的容量等。这样,本申请实施例可以无需更新终端设备、应用程序和上报组件,就配置上报组件的实施方案,从而达到控制上报流程,规避上报本身问题造成事故的目的。
例如,上报组件可以接收配置服务器配置的缓存容量,通过将所述缓存队列中数据报文的数据量与所述缓存容量进行比较,确定所述缓存队列中是否存在空闲位置。响应于所述缓存队列中数据报文的数据量小于所述缓存容量,允许向所述缓存队列写入数据。响应于所述缓存队列中数据报文的数据量不小于所述缓存容量,拒绝向所述缓存队列写入数据。
又例如,上报组件可以接收配置服务器配置的数据库容量,通过将所述数据库中数据报文的数据量与所述数据库容量进行比较,确定所述数据库中是否存在空闲位置。响应于所述数据库中数据报文的数据量小于所述数据库容量,允许向所述数据库写入数据。响应于所述数据库中数据报文的数据量不小于所述数据库容量,拒绝向所述数据库写入数据。
再例如,上报组件可以接收配置服务器配置的上传数据量。在对所述缓存队列进行轮询得到待上传数据报文集合时,上报组件可以将所述上传数据量作为限制条件,得到所述待上传数据报文集合。其中,所述待上传数据报文集合的数据量不大于所述上传数据量。
组包是指对业务数据进行合并和压缩,如果每一次上传到接入服务器的业务数据没有经过合并和压缩,网络利用率就会大大降低。特别是在车载环境网络较差的情况下,合并和压缩业务数据不仅可以提高网络利用率,也提高了上传成功率和上报及时性。
其中,业务数据可以为与终端设备中的应用客户端相关联的设备日志。在终端设备为车载终端时,设备日志可以包括车载终端中的日志记录。日志记录可以包括车载客户端遇到功能异常时所记录的异常问题。一些实施例中,日志记录也可以包括车载终端在行驶过程中的车速信息和里程信息。一些实施例中,日志记录还可以包括车载终端在行驶过程中的行驶轨迹。一些实施例中,日志记录还可以包括车载终端上的行车记录仪所拍摄的行车视频数据。这里不对日志记录中所记录的数据进行一一列举。
相应的,数据服务器可以为用于对日志记录进行数据处理的服务器。比如,数据服务器可以对车载客户端在使用过程中所遇到的异常问题进行汇总,为车载客户端提供异常问题所对应的解决方案。如果异常问题不能及时且准确的上传至数据服务器,那么没有办法获得足够的信息处理问题,在问题得不到及时处理的情况下,就会导致用户流失。又比如,数据服务器可以分析车载终端的车速信息和里程信息,确定车载终端是否出现超速行为,进而在车载终端出现超速行为时,为车载终端提示超速告警,以提高车载终端行驶的安全性。又比如,数据服务器可以对车载终端的行驶轨迹进行分析,并结合当前道路上的路况信息等,以实现准确且及时的车机服务、或提供自动驾驶服务,这里的车机服务可以为提供规避拥堵路段的功能。又比如,数据服务器可以通过行车记录仪所拍摄的行车视频数据,实时监测车辆的运行状态信息,使得无需从车载终端上拷贝行车视频数据,从而达到远程控制、应急指挥的目的。
在本申请中,涉及到设备日志中的异常问题、车速信息、里程信息、行驶轨迹和行车视频数据等相关的数据,当本申请以上实施例运用到具体产品或技术中时,需要获得用户许可或者同意,且相关数据的收集、使用和处理需要遵守所在国家相关法律法规和国家标准。例如,车载终端可以显示提示信息“是否记录车速信息和里程信息,且将记录到的信 息发送至服务器”,当车载终端对应的用户授权通过之后,车载终端才可以将车速信息和里程信息作为业务数据,进而上传该业务数据。
终端设备可以将业务层对应的日志文件映射至业务层的内存的虚拟地址空间。终端设备可以根据递增的方式,为业务数据分配业务标识(也称为目标业务标识),将目标业务标识写入日志文件。终端设备可以通过虚拟地址空间获取在某一时间段(也称为目标时间段)内的最大业务标识和最小业务标识。终端设备可以获取服务器在目标时间段内所统计到的去重业务标识数量。终端设备可以根据最大业务标识、最小业务标识和去重业务标识数量,生成上报成功率。其中,单调递增的目标业务标识可以保证生成的上报成功率的准确性。
服务器可以统计在目标时间段内所接收到的去重业务标识数量,进而将去重业务标识数量返回至终端设备,以使终端设备从虚拟地址空间获取在目标时间段内的最大业务标识和最小业务标识,进而终端设备可以根据最大业务标识、最小业务标识和去重业务标识数量,生成上报成功率。一些实施例中,终端设备还可以将从虚拟地址空间获取到的在目标时间段内的最大业务标识和最小业务标识发送至服务器,以使服务器统计在目标时间段内所接收到的去重业务标识数量,进而服务器可以根据最大业务标识、最小业务标识和去重业务标识数量,生成上报成功率。
上报成功率可以为针对一个数据服务器而言,也可以为针对多个数据服务器而言。换言之,去重业务标识数量可以为针对一个数据服务器所统计的,也可以为针对多个数据服务器所统计的。这样,在去重业务标识数量针对一个数据服务器所统计时,上报成功率为针对一个数据服务器而言。在去重业务标识数据针对全部数据服务器所统计时,上报成功率为针对全部数据服务器而言。其中,上报成功率在针对全部数据服务器而言时,可以等同于针对接入服务器而言,即去重业务标识数据可以为针对接入服务器所统计的。
由于网络不稳定等多种原因,数据服务器不可避免的会接收到重复的目标业务标识,这样,数据服务器就需要对接收到的目标业务标识进行去重,得到去重后的目标业务标识,进而将去重后的目标业务标识的数量作为去重业务标识数量。
本申请实施例可以将目标业务标识保存到mmap(memory mapping,一种内存映射文件的方法)映射的文件(即日志文件),这块内存具有在进程崩溃时会回写到文件系统的特点,既保证了进程重启后目标业务标识可以继续增长,也避免应用程序崩溃时目标业务标识来不及写入文件,导致目标业务标识不单调连续。换言之,终端设备可以将应用客户端在磁盘中的日志文件映射进应用客户端在内存中的虚拟地址空间,这个日志文件可以用于存储唯一表示的目标业务标识,在应用客户端的进程崩溃时,终端设备会释放内存,内存中的这个文件中的目标业务标识会被写回到磁盘中,从而保证目标业务标识的唯一性。同理,在进程不崩溃时,目标业务标识也会被回写到磁盘中。
其中,不同的应用客户端可以通过mmap方法映射有不同的日志文件,以使每个应用客户端会在各自的日志文件中从0开始递增表示各自的目标业务标识。
其中,终端设备根据最大业务标识、最小业务标识和去重业务标识数量,生成上报成功率的具体过程可以参见下述公式(1):
Figure PCTCN2023071183-appb-000001
其中,logid可以表示目标业务标识,distinct logid可以表示对接入服务器在目标时 间段内接收到的logid进行去重,count(distinct logid)可以表示logid去重后的总数(即去重业务标识数量);max(logid)可以表示在目标时间段内分配的最大目标业务标识,min(logid)可以表示在目标时间段内分配的最小目标业务标识,((max(logid)-min(logid))+1)可以表示在目标时间段内生成的目标业务标识的数量(即累积业务标识数量)。
终端设备可以根据最大业务标识和最小业务标识,生成累积业务标识数量,进而根据去重业务标识数量和累积业务标识数量的比值,生成上报成功率。
终端设备可以确定设备缓存队列中是否存在空闲位置,进而在缓存队列中存在空闲位置时,执行步骤S202,或者在缓存队列中不存在空闲位置时,执行步骤S203-步骤S204。
步骤S202,响应于缓存队列中存在空闲位置,则通过上报组件将第三数据报文添加至缓存队列。
一些实施例中,终端设备可以通过上报线程将第三数据报文添加至缓存队列。
业务层上报的业务数据并不会立即上传至接入层,而是会先存入缓存队列。在终端环境下(特别是车载环境下),系统资源会因为各种原因受到限制,网络随时都可能进入不稳定状态,导致数据上传成功的概率偏低。因此,将组包后的业务数据(即第三数据报文)加入缓存队列,一方面可以保存数据,失败后重试以提高数据报文的上报成功率,另一方面可以降低了磁盘访问频率,进而提高了数据报文的上传效率。
可以理解的是,通常情况下,终端设备会将待添加数据报文集合中的数据报文和第三数据报文添加至缓存队列的末尾,以在后续的轮询中从缓存队列中读取第三数据报文。其中,终端设备设备可以在后续的第一次对缓存队列进行轮询时,读取到第三数据报文,也可以在后续的第二次对缓存队列进行轮询时,读取到第三数据报文,还可以在后续的其他轮询时,读取到第三数据报文,本申请对此不做限定。
一些实施例中,终端设备还可以根据第三数据报文的报文优先级,将第三数据报文添加至缓存队列中的其他位置。例如,首部位置,这样,终端设备就可以在后续的第一次对缓存队列进行轮询时,读取到缓存队列中的第三数据报文。可以理解的是,通过为第三数据报文设置报文优先级,可以优先轮询缓存队列中的报文优先级更高的数据报文,进而优先上传报文优先级更高的数据报文,这里的报文优先级可以表示数据报文的重要程度,从而实现优先上传重要程度更高的数据报文的目的。
步骤S203,若缓存队列中不存在空闲位置,则通过上报组件将当前时间戳确定为第三数据报文的第三入库时间戳,将第三数据报文和第三入库时间戳发送至数据库线程。
步骤S204,通过数据库线程将第三数据报文和第三入库时间戳关联存储至数据库。
图6是本申请实施例提供的一种进行数据上报的流程示意图。如图6所示,业务层(即应用程序)可以执行步骤S61,通过步骤S61进行初始化,进而开始通过步骤S62启动轮询。轮询可以包含两个部分,一个部分是对数据库和缓存队列进行轮询,另一部分是每隔W秒轮询发送心跳业务数据。其中,心跳业务数据是指定期上报的一个业务数据,可以证明业务端还是在正常工作,用于数据后台分析业务端的运行情况。心跳业务数据的数据类型可以为可丢弃类型。
第一部分轮询的时间间隔可以为Y秒。Y可以为正数,例如,5秒。W可以表示第二部分轮询的时间间隔,这里的W可以为正数,例如,30秒。
如图6所示,终端设备可以执行步骤S64,通过步骤S64每隔W秒轮询发送心跳业务数据。如图6所示,终端设备在对数据库和缓存队列进行轮询时,可以首先执行步骤S63,在步骤S63中读取数据库中存储的业务数据,进而执行步骤S65,通过步骤S65对数据库中存储的业务数据进行轮询。每一次轮询可以从数据库中读取K个业务数据,将K个业务数据作为待添加数据报文集合,将待添加数据报文集合加入缓存队列。其中,这里的K可以为正整数。
终端设备可以执行步骤S66,通过步骤S66对缓存队列进行轮询,从缓存队列中遍历得到S个数据报文,将S个数据报文作为待上传数据报文集合,进而在网络连接的条件下,通过步骤S67对待上传数据报文进行上传处理,即通过网络组件对从缓存队列中取出的数据报文进行上传处理,将从缓存队列中取出的数据报文上传至接入服务器。S可以为正整数。S可以为上报组件所设定的同一时间进行上传处理的数据报文的数量。通过设定同一时间进行上传处理的数据报文的数量,可以避免网络组件占用过多的网络资源。
如图6所示,在步骤S68中,响应于待上传数据报文集合中的数据报文上传失败,则终端设备可以累积该上传失败的数据报文的连续上传失败次数,进而通过步骤S70确定连续上传失败次数是否超过次数阈值。在连续上传失败次数超过次数阈值时,终端设备可以执行步骤S71,将该上传失败的数据报文写入数据库,在之后的轮询中读取且重试。一些实施例中,在连续上传失败次数未超过次数阈值时,终端设备可以执行步骤S71,将该上传失败的数据报文重新加入缓存队列,在之后的轮询中重试。
数据报文的数据类型可以划分为可丢弃类型和不可丢弃类型两种类型。终端设备在执行步骤S71,通过步骤S71将上传失败的数据报文写入数据库时,需要确定数据报文的数据类型,进而在数据报文的数据类型为不可丢弃类型时,通过步骤S71将上传失败的数据报文写入数据库。一些实施例中,在数据报文的数据类型为可丢弃类型时,终端设备无需通过步骤S71将上传失败的数据报文写入数据库,而是直接丢弃该上传失败的数据报文。
终端设备在执行步骤S72时,可以确定数据库中的数据报文的报文数量是否超出数量阈值,进而在报文数量未超出数量阈值时,将该上传失败的数据报文写入数据库。一些实施例中,在报文数量超出数量阈值时,终端设备可以执行步骤S73,丢弃该上传失败的数据报文。
终端设备在将上传失败的数据报文写入数据库时,会附带更新一个时间戳(即更新入库时间戳),以在每一次对数据库进行轮询时,都会从数据库中读取时间戳最久的数据报文,加入到缓存队列中继续重试。换言之,终端设备可以通过LRU(Least Recently Used,最近最少使用)对数据库进行轮询。一些实施例中,终端设备还可以通过FIFO(First Input First Output,先进先出)等算法对数据库进行轮询,本申请对此不做限定。
如图6所示,在步骤S69中,响应于待上传数据报文集合中的数据报文上传成功,则终端设备可以确定数据库中是否存在该上传成功的数据报文,进而在数据库中存在该上传成功的数据报文时,从数据库中删除该上传成功的数据报文。一些实施例中,在数据库中不存在该上传成功的数据报文时,终端设备无需执行其他操作。
如图6所示,在业务层需要上报新数据时,可以执行步骤S74,通过步骤S74上传新的业务数据,进而通过步骤S75对业务数据进行组包处理。可以理解的是,终端设备可以通过步骤S76确定业务数据的数据量(即数据大小)是否满足组包条件,进而在业务数据的数据大小满足组包条件时,执行步骤S71,通过步骤S71将组包所得到的数据报文加入 缓存队列。一些实施例中,在业务数据的数据大小不满足组包条件时,终端设备可以执行步骤S77,在步骤S77中等待业务层上传新的业务数据,直到接收到的待进行组包处理的业务数据的数据大小满足组包条件时,完成组包。
终端设备在通过步骤S71将组包所得到的数据报文加入缓存队列时,需要确定缓存队列中是否存在空闲位置,进而在缓存队列中存在空闲位置时,将组包所得到的数据报文加入缓存队列。一些实施例中,在缓存队列中不存在空闲位置时,终端设备可以执行步骤S72,通过步骤S72将组包所得到的数据报文写入数据库。
同理,终端设备在执行步骤S72时,可以确定数据库中的数据报文的报文数量是否超出数量阈值,进而在报文数量未超出数量阈值时,将该组包所得到的数据报文写入数据库。一些实施例中,在报文数量超出数量阈值时,终端设备可以执行步骤S73,丢弃该组包所得到的数据报文。
当上报专用线程(即上报线程)中的缓存队列满,上报线程需要直接将数据报文转发至数据库专用线程(即数据库线程)。当上报线程上报失败(即连续上传失败次数超过次数阈值)时,上报线程需要间接将数据报文转发至数据库线程,以使数据库线程执行步骤S72,将数据报文写入数据库。同理,当轮询时从数据库读取数据时,数据库线程需要从数据库中读取数据报文,进而将数据报文转发至上报线程,以使上报线程执行步骤S71,将数据报文加入缓存队列。
终端设备可以在任意时刻执行上述步骤S201-步骤S204。当执行上述步骤S202时,上述图3所对应实施例的缓存队列中可以包含第三数据报文。在对缓存队列进行轮询时,有机会将第三数据报文作为待上传数据报文集合。当执行上述步骤S203-步骤S204时,上述图3所对应实施例的数据库中可以包含第三数据报文。在对数据库进行轮询时,有机会将第三数据报文作为待添加数据报文集合。
由此可见,本申请实施例可以在获取到待上传的业务数据之后,对业务数据进行组包处理,进而将组包得到的第三数据报文添加至缓存队列或数据库中,在后续对缓存队列或数据库的轮询过程中轮询得到第三数据报文,以实现第三数据报文中的业务数据的上传处理。可以理解的是,通过对业务数据的组包处理,可以提高网络利用率,且提高数据报文的上传成功率和上报及时性。
图7是本申请实施例提供的一种数据处理装置的结构示意图,该数据处理装置1可以包括:数据库轮询模块11,队列轮询模块12,数据添加模块13,数据删除模块14。该数据处理装置1还可以包括:次数累积模块15,第一比较模块16,第二比较模块17,数据生成模块18,第一处理模块19,第二处理模块20。
数据库轮询模块11,用于对数据库中的数据报文进行轮询,得到待添加数据报文集合,将待添加数据报文集合添加至缓存队列;
数据库轮询模块11包括:数据库轮询单元111,集合添加单元112。
数据库轮询单元111,用于响应于缓存队列中存在空闲位置,则通过上报组件调用数据库线程对数据库中的数据报文进行轮询,得到待添加数据报文集合。
一些实施例中,数据库轮询单元111,用于响应于缓存队列中存在空闲位置,则通过上报组件调用数据库线程,在数据库线程中,根据数据库中的数据报文的第一入库时间戳,对数据库中的数据报文进行排序,得到排序后的数据报文。
一些实施例中,数据库轮询单元111,用于从排序后的数据报文中轮询得到K个数据 报文,将K个数据报文确定为待添加数据报文集合;K为正整数。
集合添加单元112,用于通过数据库线程将待添加数据报文集合发送至上报组件,通过上报组件将待添加数据报文集合添加至缓存队列。
数据库轮询单元111和集合添加单元112的具体实现方式,可以参见上述图3所对应实施例中对步骤S101的描述,这里将不再进行赘述。
队列轮询模块12,用于从缓存队列中轮询得到待上传数据报文集合,对待上传数据报文集合进行上传处理。
一些实施例中,队列轮询模块12,用于响应于缓存队列中存在数据报文,则在上报组件中,根据缓存队列中的数据报文的队列位置,从缓存队列中轮询得到S个数据报文,将S个数据报文确定为待上传数据报文集合。S为正整数。
一些实施例中,队列轮询模块12,用于基于网络组件对待上传数据报文集合进行上传处理。
数据添加模块13,用于响应于待上传数据报文集合中的第一数据报文满足连续上传失败条件,则将第一数据报文添加至数据库。
一些实施例中,数据添加模块13,用于响应于待上传数据报文集合中的第一数据报文满足连续上传失败条件,则获取第一数据报文的数据类型;
一些实施例中,数据添加模块13,用于响应于数据类型为不可丢弃类型,且数据库中的数据报文的报文数量小于数量阈值,且数据库中不存在第一数据报文,则通过上报组件将当前时间戳确定为第一数据报文的第二入库时间戳,将第一数据报文和第二入库时间戳发送至数据库线程,通过数据库线程将第一数据报文和第二入库时间戳关联存储至数据库。
一些实施例中,数据添加模块13,还具体用于响应于数据类型为可丢弃类型,则通过上报组件删除第一数据报文。
一些实施例中,数据添加模块13,还用于响应于数据类型为不可丢弃类型,且数据库中的数据报文的报文数量小于数量阈值,且数据库中存在第一数据报文,则通过上报组件获取第一数据报文的更新入库时间戳,将第一数据报文和更新入库时间戳发送至数据库线程,通过数据库线程将第一数据报文在数据库中的第二入库时间戳更新为更新入库时间戳。
一些实施例中,数据添加模块13,还用于响应于数据类型为不可丢弃类型,且数据库中的数据报文的报文数量大于或等于数量阈值,则通过上报组件删除第一数据报文。
数据删除模块14,用于响应于待上传数据报文集合中的第二数据报文成功上传至服务器,且数据库中存在第二数据报文,则从数据库中删除第二数据报文。
一些实施例中,次数累积模块15,用于响应于待上传数据报文集合中的第一数据报文上传失败,则累计第一数据报文的连续上传失败次数;
第一比较模块16,用于响应于连续上传失败次数达到次数阈值,则确定第一数据报文满足连续上传失败条件;
第二比较模块17,用于响应于连续上传失败次数未达到次数阈值,则确定第一数据报文不满足连续上传失败条件,通过上报组件将第一数据报文重新添加至缓存队列。
一些实施例中,数据生成模块18,用于获取业务层上传的业务数据,对业务数据进行组包处理,生成第三数据报文。
一些实施例中,数据生成模块18,用于在业务线程中,获取业务层上传的业务数据,通过业务线程将业务数据发送至上报组件。
一些实施例中,数据生成模块18,用于响应于业务数据的数据量满足组包条件,则基于上报组件对业务数据进行组包处理,生成第三数据报文。
第一处理模块19,用于响应于缓存队列中存在空闲位置,则通过上报组件将第三数据报文添加至缓存队列。
一些实施例中,第二处理模块20,用于响应于缓存队列中不存在空闲位置,则通过上报组件将当前时间戳确定为第三数据报文的第三入库时间戳,将第三数据报文和第三入库时间戳发送至数据库线程。
第二处理模块20,用于通过数据库线程将第三数据报文和第三入库时间戳关联存储至数据库。
一些实施例中,数据处理装置1还用于将业务层对应的日志文件映射至业务层的内存的虚拟地址空间。
一些实施例中,数据处理装置1还用于根据递增的方式,为业务数据分配目标业务标识,将目标业务标识写入日志文件。
一些实施例中,数据处理装置1还用于通过虚拟地址空间获取在目标时间段内的最大业务标识和最小业务标识。
一些实施例中,数据处理装置1还用于获取服务器在目标时间段内所统计到的去重业务标识数量。
一些实施例中,数据处理装置1还用于根据最大业务标识、最小业务标识和去重业务标识数量,生成上报成功率。
一些实施例中,数据处理装置1还用于响应于待上传数据报文集合中的第二数据报文成功上传至接入层,则接入层用于对第二数据报文进行解包处理,将解包得到的业务数据分发至服务器。
其中,数据库轮询模块11,队列轮询模块12,数据添加模块13和数据删除模块14的具体实现方式,可以参见上述图3所对应实施例中对步骤S101-步骤S104的描述,这里将不再进行赘述。其中,次数累积模块15,第一比较模块16,第二比较模块17,数据生成模块18,第一处理模块19和第二处理模块20的具体实现方式,可以参见上述图5所对应实施例中对步骤S201-步骤S204的描述,这里将不再进行赘述。另外,对采用相同方法的有益效果描述,也不再进行赘述。
一些实施例中,数据处理装置1还可以包括配置接收模块,用于接收配置服务器的配置信息,所述配置信息包括以下之一:
缓存容量,用于指定所述缓存队列允许写入数据量的最大值;
数据库容量,用于指定所述数据库允许写入数据量的最大值;
上传数据量,用于指定同时上传的数据报文的最大数据量。
数据处理装置1可以通过将所述缓存队列中数据报文的数据量与所述缓存容量进行比较,确定所述缓存队列中是否存在空闲位置。响应于所述缓存队列中数据报文的数据量小于所述缓存容量,允许向所述缓存队列写入数据。响应于所述缓存队列中数据报文的数据量不小于所述缓存容量,拒绝向所述缓存队列写入数据。
数据处理装置1可以通过将所述数据库中数据报文的数据量与所述数据库容量进行比较,确定所述数据库中是否存在空闲位置。响应于所述数据库中数据报文的数据量小于所述数据库容量,允许向所述数据库写入数据。响应于所述数据库中数据报文的数据量不小 于所述数据库容量,拒绝向所述数据库写入数据。
队列轮询模块12在对所述缓存队列进行轮询得到待上传数据报文集合时,可以将所述上传数据量作为限制条件,得到所述待上传数据报文集合。其中,所述待上传数据报文集合的数据量不大于所述上传数据量。
图8是本申请实施例提供的一种计算机设备的结构示意图。如图8所示,该计算机设备1000可以包括:处理器1001,网络接口1004和存储器1005。此外,上述计算机设备1000还可以包括:用户接口1003,和至少一个通信总线1002。其中,通信总线1002用于实现这些组件之间的连接通信。在一些实施例中,用户接口1003可以包括显示屏(Display)、键盘(Keyboard),可选用户接口1003还可以包括标准的有线接口、无线接口。一些实施例中,网络接口1004可以包括标准的有线接口、无线接口(如WI-FI接口)。存储器1005可以是高速RAM存储器,也可以是非不稳定的存储器(non-volatile memory),例如至少一个磁盘存储器。一些实施例中,存储器1005还可以是至少一个位于远离前述处理器1001的存储装置。如图8所示,作为一种计算机可读存储介质的存储器1005中可以包括操作系统、网络通信模块、用户接口模块以及设备控制应用程序。
在如图8所示的计算机设备1000中,网络接口1004可提供网络通讯功能;而用户接口1003主要用于为用户提供输入的接口;而处理器1001可以用于调用存储器1005中存储的设备控制应用程序,以实现:
对数据库中的数据报文进行轮询,得到待添加数据报文集合,将待添加数据报文集合添加至缓存队列;
从缓存队列中轮询得到待上传数据报文集合,对待上传数据报文集合进行上传处理;
响应于待上传数据报文集合中的第一数据报文满足连续上传失败条件,则将第一数据报文添加至数据库;
响应于待上传数据报文集合中的第二数据报文成功上传至服务器,且数据库中存在第二数据报文,则从数据库中删除第二数据报文。
应当理解,本申请实施例中所描述的计算机设备1000可执行前文图3和图5所对应实施例中对数据处理方法的描述,也可执行前文图7所对应实施例中对数据处理装置1的描述,在此不再赘述。另外,对采用相同方法的有益效果描述,也不再进行赘述。
此外,这里需要指出的是:本申请实施例还提供了一种计算机可读存储介质,且计算机可读存储介质中存储有前文提及的数据处理装置1所执行的计算机程序,且计算机程序包括程序指令,当处理器执行程序指令时,能够执行前文图3和图5所对应实施例中对数据处理方法的描述,因此,这里将不再进行赘述。另外,对采用相同方法的有益效果描述,也不再进行赘述。对于本申请所涉及的计算机可读存储介质实施例中未披露的技术细节,请参照本申请方法实施例的描述。
此外,需要说明的是:本申请实施例还提供了一种计算机程序产品或计算机程序,该计算机程序产品或者计算机程序可以包括计算机指令,该计算机指令可以存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令,处理器可以执行该计算机指令,使得该计算机设备执行前文图3和图5所对应实施例中对数据处理方法的描述,因此,这里将不再进行赘述。另外,对采用相同方法的有益效果描述,也不再进行赘述。对于本申请所涉及的计算机程序产品或者计算机程序实施例中未披露的技术 细节,请参照本申请方法实施例的描述。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程,是可以通过计算机程序来指令相关的硬件来完成,计算机程序可存储于一计算机可读取存储介质中,该程序在执行时,可包括如上述各方法的实施例的流程。其中,存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory,ROM)或随机存储记忆体(Random Access Memory,RAM)等。
以上实施例的各技术特征可以进行任意的组合,为使描述简洁,未对上述实施例中的各个技术特征所有可能的组合都进行描述,然而,只要这些技术特征的组合不存在矛盾,都应当认为是本说明书记载的范围。
以上所揭露的仅为本申请较佳实施例而已,当然不能以此来限定本申请之权利范围,因此依本申请权利要求所作的等同变化,仍属本申请所涵盖的范围。

Claims (20)

  1. 一种数据处理方法,应用于车载中控设备,所述车载中控设备中设置有数据库和缓存队列,所述方法包括:
    对所述数据库中的数据报文进行轮询,得到待添加数据报文集合,将所述待添加数据报文集合添加至所述缓存队列;
    对所述缓存队列进行轮询得到待上传数据报文集合,将所述待上传数据报文集合通过所述车载中控设备的无线通信连接进行上传;
    响应于所述待上传数据报文集合中的第一数据报文满足连续上传失败条件,则将所述第一数据报文存储在所述数据库中;
    响应于所述待上传数据报文集合中的第二数据报文成功上传至服务器,且所述数据库中存在所述第二数据报文,则从所述数据库中删除所述第二数据报文。
  2. 根据权利要求1所述的方法,进一步包括:
    接收配置服务器配置的缓存容量;
    通过将所述缓存队列中数据报文的数据量与所述缓存容量进行比较,确定所述缓存队列中是否存在空闲位置;
    响应于所述缓存队列中数据报文的数据量小于所述缓存容量,允许向所述缓存队列写入数据;
    响应于所述缓存队列中数据报文的数据量不小于所述缓存容量,拒绝向所述缓存队列写入数据。
  3. 根据权利要求1所述的方法,进一步包括:
    接收配置服务器配置的数据库容量;
    通过将所述数据库中数据报文的数据量与所述数据库容量进行比较,确定所述数据库中是否存在空闲位置;
    响应于所述数据库中数据报文的数据量小于所述数据库容量,允许向所述数据库写入数据;
    响应于所述数据库中数据报文的数据量不小于所述数据库容量,拒绝向所述数据库写入数据。
  4. 根据权利要求1所述的方法,进一步包括:
    接收配置服务器配置的上传数据量;
    其中,对所述缓存队列进行轮询得到待上传数据报文集合包括:
    将所述上传数据量作为限制条件,得到所述待上传数据报文集合,其中,所述待上传数据报文集合的数据量不大于所述上传数据量。
  5. 根据权利要求1所述的方法,其中,所述对数据库中的数据报文进行轮询,得到待添加数据报文集合,将所述待添加数据报文集合添加至缓存队列,包括:
    响应于缓存队列中存在空闲位置,则通过上报组件调用数据库线程对数据库中的数据 报文进行轮询,得到待添加数据报文集合;
    通过所述数据库线程将所述待添加数据报文集合发送至所述上报组件,通过所述上报组件将所述待添加数据报文集合添加至所述缓存队列。
  6. 根据权利要求5所述的方法,其中,所述响应于缓存队列中存在空闲位置,则通过上报组件调用数据库线程对数据库中的数据报文进行轮询,得到待添加数据报文集合,包括:
    响应于缓存队列中存在空闲位置,则通过上报组件调用数据库线程,在所述数据库线程中,根据数据库中的数据报文的第一入库时间戳,对所述数据库中的数据报文进行排序,得到排序后的数据报文;
    从所述排序后的数据报文中轮询得到K个数据报文,将所述K个数据报文确定为待添加数据报文集合;所述K为正整数。
  7. 根据权利要求1所述的方法,其中,所述从所述缓存队列中轮询得到待上传数据报文集合,对所述待上传数据报文集合进行上传处理,包括:
    响应于所述缓存队列中存在数据报文,则在上报组件中,根据所述缓存队列中的数据报文的队列位置,从所述缓存队列中轮询得到S个数据报文,将所述S个数据报文确定为待上传数据报文集合;所述S为正整数;
    基于网络组件对所述待上传数据报文集合进行上传处理。
  8. 根据权利要求1所述的方法,进一步包括:
    响应于所述待上传数据报文集合中的所述第一数据报文上传失败,则累计所述第一数据报文的连续上传失败次数;
    响应于所述连续上传失败次数达到次数阈值,则确定所述第一数据报文满足连续上传失败条件;
    响应于所述连续上传失败次数未达到所述次数阈值,则确定所述第一数据报文不满足所述连续上传失败条件,通过上报组件将所述第一数据报文重新添加至所述缓存队列。
  9. 根据权利要求1所述的方法,其中,所述响应于所述待上传数据报文集合中的第一数据报文满足连续上传失败条件,则将所述第一数据报文添加至所述数据库,包括:
    响应于所述待上传数据报文集合中的第一数据报文满足连续上传失败条件,则获取所述第一数据报文的数据类型;
    响应于所述数据类型为不可丢弃类型,且所述数据库中的数据报文的报文数量小于数量阈值,且所述数据库中不存在所述第一数据报文,则通过上报组件将当前时间戳确定为所述第一数据报文的第二入库时间戳,将所述第一数据报文和所述第二入库时间戳发送至数据库线程,通过所述数据库线程将所述第一数据报文和所述第二入库时间戳关联存储至所述数据库。
  10. 根据权利要求9所述的方法,进一步包括:
    响应于所述数据类型为可丢弃类型,则通过所述上报组件删除所述第一数据报文;
    响应于所述数据类型为所述不可丢弃类型,且所述数据库中的数据报文的所述报文数量小于所述数量阈值,且所述数据库中存在所述第一数据报文,则通过所述上报组件获取所述第一数据报文的更新入库时间戳,将所述第一数据报文和所述更新入库时间戳发送至所述数据库线程,通过所述数据库线程将所述第一数据报文在所述数据库中的所述第二入库时间戳更新为所述更新入库时间戳;
    响应于所述数据类型为所述不可丢弃类型,且所述数据库中的数据报文的所述报文数量大于或等于所述数量阈值,则通过所述上报组件删除所述第一数据报文。
  11. 根据权利要求1所述的方法,进一步包括:
    获取业务层上传的业务数据,对所述业务数据进行组包处理,生成第三数据报文;
    响应于所述缓存队列中存在空闲位置,则通过上报组件将所述第三数据报文添加至所述缓存队列。
  12. 根据权利要求11所述的方法,其中,所述获取业务层上传的业务数据,对所述业务数据进行组包处理,生成第三数据报文,包括:
    在业务线程中,获取业务层上传的业务数据,通过所述业务线程将所述业务数据发送至上报组件;
    响应于所述业务数据的数据量满足组包条件,则基于所述上报组件对所述业务数据进行组包处理,生成第三数据报文。
  13. 根据权利要求11所述的方法,进一步包括:
    响应于所述缓存队列中不存在所述空闲位置,则通过所述上报组件将当前时间戳确定为所述第三数据报文的第三入库时间戳,将所述第三数据报文和所述第三入库时间戳发送至数据库线程;
    通过所述数据库线程将所述第三数据报文和所述第三入库时间戳关联存储至所述数据库。
  14. 根据权利要求11所述的方法,进一步包括:
    将所述业务层对应的日志文件映射至所述业务层的内存的虚拟地址空间;
    根据递增的方式,为所述业务数据分配目标业务标识,将所述目标业务标识写入所述日志文件;
    通过所述虚拟地址空间获取在目标时间段内的最大业务标识和最小业务标识;
    获取所述服务器在所述目标时间段内所统计到的去重业务标识数量;
    根据所述最大业务标识、所述最小业务标识和所述去重业务标识数量,生成上报成功率。
  15. 根据权利要求1所述的方法,进一步包括:
    响应于所述待上传数据报文集合中的第二数据报文成功上传至接入层,则所述接入层用于对所述第二数据报文进行解包处理,将解包得到的业务数据分发至所述服务器。
  16. 一种数据处理装置,设置于车载中控设备中,所述车载中控设备中设置有数据库和缓存队列,所述装置包括:
    数据库轮询模块,用于对数据库中的数据报文进行轮询,得到待添加数据报文集合,将所述待添加数据报文集合添加至缓存队列;
    队列轮询模块,用于对所述缓存队列进行轮询得到待上传数据报文集合,通过所述车载中控设备的无线通信连接将所述待上传数据报文集合进行上传;
    数据添加模块,用于响应于所述待上传数据报文集合中的第一数据报文满足连续上传失败条件,则将所述第一数据报文添加至所述数据库;
    数据删除模块,用于响应于所述待上传数据报文集合中的第二数据报文成功上传至服务器,且所述数据库中存在所述第二数据报文,则从所述数据库中删除所述第二数据报文。
  17. 根据权利要求16所述的数据处理装置,进一步包括:
    配置接收模块,用于接收配置服务器的配置信息,所述配置信息包括以下之一:
    缓存容量,用于指定所述缓存队列允许写入数据量的最大值;
    数据库容量,用于指定所述数据库允许写入数据量的最大值;
    上传数据量,用于指定同时上传的数据报文的最大数据量。
  18. 一种计算机设备,其特征在于,包括:处理器和存储器;
    所述处理器与所述存储器相连,其中,所述存储器用于存储计算机程序,所述处理器用于调用所述计算机程序,以使得所述计算机设备执行权利要求1-15任一项所述的方法。
  19. 一种计算机可读存储介质,其特征在于,所述计算机可读存储介质中存储有计算机程序,该计算机程序适于由处理器加载并执行,以使得具有所述处理器的计算机设备执行权利要求1-15任一项所述的方法。
  20. 一种计算机程序产品,其特征在于,所述计算机程序产品包括计算机指令,该计算机指令存储在计算机可读存储介质中,且适于由处理器读取并执行,以使得具有所述处理器的计算机设备执行权利要求1-15任一项所述的方法。
PCT/CN2023/071183 2022-02-25 2023-01-09 一种数据处理方法、装置、计算机设备以及可读存储介质 Ceased WO2023160276A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP23758901.5A EP4407957B1 (en) 2022-02-25 2023-01-09 Data processing method and apparatus, computer device and readable storage medium
US18/390,299 US20240129251A1 (en) 2022-02-25 2023-12-20 Data processing method and apparatus, computer device, and readable storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210180680.4A CN114567674B (zh) 2022-02-25 2022-02-25 一种数据处理方法、装置、计算机设备以及可读存储介质
CN202210180680.4 2022-02-25

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/390,299 Continuation US20240129251A1 (en) 2022-02-25 2023-12-20 Data processing method and apparatus, computer device, and readable storage medium

Publications (1)

Publication Number Publication Date
WO2023160276A1 true WO2023160276A1 (zh) 2023-08-31

Family

ID=81716473

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/071183 Ceased WO2023160276A1 (zh) 2022-02-25 2023-01-09 一种数据处理方法、装置、计算机设备以及可读存储介质

Country Status (4)

Country Link
US (1) US20240129251A1 (zh)
EP (1) EP4407957B1 (zh)
CN (1) CN114567674B (zh)
WO (1) WO2023160276A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119544825A (zh) * 2024-11-07 2025-02-28 重庆赛力斯凤凰智创科技有限公司 车载数据采集上传方法、系统、电子设备及存储介质

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114567674B (zh) * 2022-02-25 2024-03-15 腾讯科技(深圳)有限公司 一种数据处理方法、装置、计算机设备以及可读存储介质
CN117648288B (zh) * 2022-09-05 2024-09-17 华为技术有限公司 文件处理方法及电子设备
CN119618082A (zh) * 2025-02-13 2025-03-14 华南农业大学 一种便携式动植物表型测定方法、系统及装置

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009284085A (ja) * 2008-05-20 2009-12-03 Autonetworks Technologies Ltd 中継装置及び中継方法
CN106878966A (zh) * 2017-02-21 2017-06-20 中国联合网络通信集团有限公司 短信下发方法、短信中心和短信发送系统
CN110493116A (zh) * 2018-05-14 2019-11-22 广州小鹏汽车科技有限公司 一种车联网数据传输方法及装置
CN110764936A (zh) * 2019-10-28 2020-02-07 深圳前海微众银行股份有限公司 一种数据采集方法及装置
CN113965482A (zh) * 2021-10-19 2022-01-21 北京天融信网络安全技术有限公司 基于gRPC的数据传输方法、装置及存储介质
CN114567674A (zh) * 2022-02-25 2022-05-31 腾讯科技(深圳)有限公司 一种数据处理方法、装置、计算机设备以及可读存储介质

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106970846B (zh) * 2017-03-15 2019-08-16 银清科技(北京)有限公司 支付系统消息控制和处理方法、装置
WO2020014954A1 (zh) * 2018-07-20 2020-01-23 威富通科技有限公司 数据控制方法及终端设备
CN109299120A (zh) * 2018-08-31 2019-02-01 深圳市商汤科技有限公司 数据处理方法和装置、电子设备、计算机存储介质
CN111355673A (zh) * 2018-12-24 2020-06-30 深圳市中兴微电子技术有限公司 一种数据处理方法、装置、设备及存储介质
CN111694766B (zh) * 2020-06-16 2024-03-08 北京经纬恒润科技股份有限公司 一种can数据采集方法及装置
CN113938258A (zh) * 2020-06-29 2022-01-14 中兴通讯股份有限公司 数据传输方法、装置以及计算机存储介质
CN112118167B (zh) * 2020-08-17 2022-04-12 深圳市风云实业有限公司 一种跨网隧道数据快速传输方法
CN113242165B (zh) * 2021-03-31 2023-04-28 惠州市德赛西威汽车电子股份有限公司 一种新能源车辆国标数据处理及上传方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009284085A (ja) * 2008-05-20 2009-12-03 Autonetworks Technologies Ltd 中継装置及び中継方法
CN106878966A (zh) * 2017-02-21 2017-06-20 中国联合网络通信集团有限公司 短信下发方法、短信中心和短信发送系统
CN110493116A (zh) * 2018-05-14 2019-11-22 广州小鹏汽车科技有限公司 一种车联网数据传输方法及装置
CN110764936A (zh) * 2019-10-28 2020-02-07 深圳前海微众银行股份有限公司 一种数据采集方法及装置
CN113965482A (zh) * 2021-10-19 2022-01-21 北京天融信网络安全技术有限公司 基于gRPC的数据传输方法、装置及存储介质
CN114567674A (zh) * 2022-02-25 2022-05-31 腾讯科技(深圳)有限公司 一种数据处理方法、装置、计算机设备以及可读存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4407957A4 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119544825A (zh) * 2024-11-07 2025-02-28 重庆赛力斯凤凰智创科技有限公司 车载数据采集上传方法、系统、电子设备及存储介质

Also Published As

Publication number Publication date
EP4407957A1 (en) 2024-07-31
EP4407957A4 (en) 2025-04-02
US20240129251A1 (en) 2024-04-18
EP4407957B1 (en) 2026-03-25
CN114567674A (zh) 2022-05-31
CN114567674B (zh) 2024-03-15

Similar Documents

Publication Publication Date Title
WO2023160276A1 (zh) 一种数据处理方法、装置、计算机设备以及可读存储介质
US11836533B2 (en) Automated reconfiguration of real time data stream processing
US9984013B2 (en) Method, controller, and system for service flow control in object-based storage system
US10761752B1 (en) Memory pool configuration for allocating memory in a distributed network
CN112433920B (zh) 一种埋点数据上报方法、装置、设备及介质
CN104601696A (zh) 服务处理方法、服务调用框架、装置和架构
CN110430142B (zh) 用于控制流量的方法和装置
CN113342746A (zh) 文件管理系统、文件管理方法、电子设备和存储介质
WO2023116438A1 (zh) 一种数据访问方法、装置以及设备
WO2023142605A1 (zh) 一种基于区块链的数据处理方法和相关装置
CN109298937A (zh) 文件解析方法及网络设备
CN107615259B (zh) 一种数据处理方法及系统
CN114048059A (zh) 接口的超时时间调整方法、装置、计算机设备及存储介质
CN110650182B (zh) 网络缓存方法、装置、计算机设备及存储介质
CN117978658A (zh) Kubernetes集群的容器伸缩系统、方法和设备
CN118585404A (zh) 数据采集方法、数据库服务器及存储介质
US9893972B1 (en) Managing I/O requests
CN111723115A (zh) 基于多级缓存的数据统计方法、装置、终端及存储介质
CN115587390A (zh) 一种公有云对象存储的实现方法
CN114500403A (zh) 一种数据处理方法、装置及计算机可读存储介质
JP2025540743A (ja) データ伝送方法、装置、伝送機器、端末機器、及びコンピュータプログラム
HK40071914B (zh) 一种数据处理方法、装置、计算机设备以及可读存储介质
HK40071914A (zh) 一种数据处理方法、装置、计算机设备以及可读存储介质
CN108121580B (zh) 应用程序通知服务的实现方法及装置
CN115408363A (zh) 数据处理方法、可读介质和电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23758901

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023758901

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2023758901

Country of ref document: EP

Effective date: 20240422

NENP Non-entry into the national phase

Ref country code: DE

WWG Wipo information: grant in national office

Ref document number: 2023758901

Country of ref document: EP