CN118819899A - A data processing method, medium and electronic device for financial transaction scenarios - Google Patents

Abstract

The present invention discloses a data processing method, medium and electronic device for financial transaction scenarios, the method comprising: initializing a ring buffer; using ThreadLocal for thread local storage; in a producer thread, obtaining a thread local event object through ThreadLocal; obtaining the next available sequence number of the ring buffer; writing the data in the thread local event object into the event object in the ring buffer; publishing events through the ring buffer to notify consumers that there are new events to be processed; the consumer thread obtains events from the ring buffer; using a thread local processing status object to process transaction data obtained from the ring buffer; after processing the transaction data, updating the thread local processing status object. The present invention provides an efficient, low-latency, thread-safe financial transaction system data processing method, which meets the strict requirements of high concurrency, high throughput and low latency of the financial system.

Description

A data processing method, medium and electronic device for financial transaction scenarios

Technical Field

The present invention relates to the field of financial transactions, and in particular to a data processing method, a medium and an electronic device for use in financial transaction scenarios.

Background Art

In high-concurrency data processing scenarios, such as online trading systems and real-time data analysis, the system needs to process a large number of concurrent requests quickly and efficiently. Currently, in high-frequency data processing scenarios, the common implementation scheme is to use traditional blocking queues (such as ArrayBlockingQueue or LinkedBlockingQueue) combined with thread pools to produce and consume data.

Traditional queue and lock mechanisms often fail to meet high performance requirements and easily lead to thread contention and performance bottlenecks. RingBuffer, as an efficient buffer structure, can provide high throughput and low latency, while ThreadLocal can provide independent variable storage for each thread to avoid sharing and contention between threads.

Summary of the invention

In view of the above problems, the present invention is proposed to provide a data processing method, medium and electronic device for financial transaction scenarios that overcome the above problems or at least partially solve the above problems.

To achieve the above-mentioned purpose, a first aspect of the present application provides a data processing method for a financial transaction scenario, the method comprising:

Initialize the ring buffer;

Use ThreadLocal for thread local storage;

In the producer thread, obtain the thread local event object through ThreadLocal and write the transaction data into the thread local event object;

Get the next available sequence number in the ring buffer, and use the obtained sequence number to get the corresponding event object from the ring buffer;

Write the data in the thread local event object to the event object in the ring buffer;

Publish events through the ring buffer to notify consumers that new events are available for processing;

The consumer thread obtains events from the ring buffer, uses the serial number of the consumer thread to identify the events to be processed, and obtains the local processing status object of the consumer thread through ThreadLocal;

Use the thread-local processing state object to process the transaction data from the ring buffer;

After processing the transaction data, update the thread-local processing status object.

Optionally, the initializing the ring buffer includes:

Select the buffer based on the expected concurrency and data processing requirements of the trading system;

Choose a wait strategy to balance system latency and CPU usage;

Create an event factory, which is used to create event objects in the ring buffer;

Initialize the Disruptor framework, create and manage the ring buffer through the Disruptor framework;

Configure the event handler, which is used to process events read from the ring buffer.

Optionally, the use of ThreadLocal for thread local storage includes:

Determine the objects that need ThreadLocal storage;

Create ThreadLocal variables for each object that requires thread local storage;

Initialize ThreadLocal variables. During system initialization, initialize ThreadLocal variables for each thread.

Get the ThreadLocal object in the thread. When processing transaction data, get the object stored locally in the thread through the ThreadLocal variable.

In the producer thread, write the transaction data into the transaction event object stored in ThreadLocal;

In the consumer thread, the processing status object is obtained through the ThreadLocal variable.

Optionally, the stored objects include transaction event objects and processing status objects, the transaction event objects include transaction ID and transaction price, and the processing status objects include intermediate states of transaction processing and results of transaction processing.

Optionally, after the transaction data is processed, logging operations may be performed or other systems may be notified based on the processing results.

Optionally, the ring buffer implements lock-free sequence number acquisition and event publishing through a Compare-And-Swap mechanism.

Optionally, the ring buffer supports multi-producer and multi-consumer mode. Through the synchronization mechanism, multiple producers can publish events at the same time, and multiple consumers can process events at the same time.

Optionally, through lock-free design and pre-allocated memory, event objects are pre-allocated in a ring buffer to ensure data consistency and processing accuracy in a high-concurrency environment.

In a second aspect of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is loaded and executed by a processor, the method described in any one of the first aspects is adopted.

In a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein when the processor loads and executes the computer program, the method described in any one of the first aspects is adopted.

In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

The present invention combines the use of a ring buffer and ThreadLocal to provide an efficient, low-latency, thread-safe financial transaction system data processing method, which significantly improves the processing performance, meets the strict requirements of high concurrency, high throughput, and low latency of the financial system, and has broad application prospects and market value.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a data processing method for a financial transaction scenario provided in an embodiment of the present application.

DETAILED DESCRIPTION

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

Referring to FIG. 1 , this embodiment provides a data processing method for a financial transaction scenario, including:

S1. Initialize the ring buffer.

In financial transaction scenarios, initializing the ring buffer (hereinafter referred to as RingBuffer) is one of the key steps to ensure that the system can efficiently handle high-concurrency transaction requests. The detailed steps are as follows:

S101. Determine the buffer size.

The buffer size is the core parameter of RingBuffer, which directly affects the throughput and performance of the system. Choose an appropriate buffer size based on the expected concurrency and data processing requirements of the trading system. For example, set the buffer size to 1024 or 2048 to ensure that there is enough buffer space in a high-concurrency environment.

S102: Select a waiting strategy.

The waiting strategy determines the behavior of the consumer when waiting for the producer to publish new data. Common waiting strategies include:

BlockingWaitStrategy: Uses locks and condition variables, suitable for scenarios where latency requirements are not high.

YieldingWaitStrategy: The consumer thread will continue to give up the CPU, which is suitable for low latency and high throughput scenarios.

BusySpinWaitStrategy: The consumer thread is busy waiting, which is suitable for extremely low latency scenarios, but consumes more CPU resources.

Choose an appropriate wait strategy to balance system latency and CPU usage.

S103. Create an event factory.

The event factory is used to create event objects in RingBuffer. Event objects in financial transaction scenarios usually contain information such as transaction ID and price.

S104. Initialize the Disruptor framework.

The Disruptor framework is the core tool for implementing RingBuffer. Through the Disruptor framework, you can easily create and manage RingBuffer, as well as producer and consumer threads.

S105. Configure event handler.

The event processor is used to process events read from the RingBuffer. Depending on the needs of the trading system, a single or multiple event processors can be configured to process trading data.

S2. Use ThreadLocal for thread local storage.

In financial trading scenarios, in order to avoid data sharing and competition between threads, we use ThreadLocal to provide independent variable storage for each thread. The detailed steps are as follows:

S201. Determine the objects that need to be stored in ThreadLocal.

In financial trading systems, determine which data or objects need to be stored locally in the thread to avoid data sharing and contention between threads. Typically, these objects include:

Transaction event object (TradeEvent): includes transaction ID, transaction price and other information.

Processing State Object: includes the intermediate state of transaction processing and the result of transaction processing.

S202. Create a ThreadLocal variable.

Create a ThreadLocal variable for each object that requires thread local storage. These variables exist independently in each thread, avoiding data competition between threads.

S203. Initialize ThreadLocal variables.

During system initialization, initialize the ThreadLocal variable for each thread. Use the withInitial method of ThreadLocal to allocate a separate object instance for each thread.

S204. Get the ThreadLocal object in the thread.

When processing transaction data, objects stored locally in the thread are obtained through ThreadLocal variables. These objects are only used within the thread, avoiding data sharing between threads.

S205. Write the data into the ThreadLocal object.

In the producer thread, the transaction data is written to the transaction event object stored in ThreadLocal. In this way, each producer thread can process its own data independently without conflicting with other threads.

S206. Process data and update ThreadLocal status.

In the consumer thread, the processing status object is obtained through the ThreadLocal variable. The object is used to process the transaction data obtained from the RingBuffer and the processing status is updated after the processing is completed.

S207. Release the ThreadLocal object.

In some cases, after the thread has finished executing, you need to manually clean up the ThreadLocal object to prevent memory leaks. This can be done by calling the ThreadLocal.remove() method.

S3, Event Production

S301. Obtain a thread-local event object.

In the producer thread, the thread-local event object is obtained through ThreadLocal. This ensures that each thread has an independent event object and avoids data sharing and competition between threads.

S302: Write transaction data.

Write the transaction data (such as transaction ID and price) to the thread-local event object. This operation is thread-local and does not involve synchronization issues between threads.

S303. Get the next sequence number of RingBuffer.

Use the `next()` method of RingBuffer to get the next available sequence number. This is a lock-free operation, which is thread-safe through the CAS (Compare-And-Swap) mechanism.

S304: Get the event object in RingBuffer.

Get the corresponding event object from the RingBuffer through the obtained sequence number. RingBuffer pre-allocates memory to ensure that the obtained event object is allocated without additional memory allocation operations.

S305: Write the data into the event object in RingBuffer.

Write the data in the thread local event object to the event object in the RingBuffer. This step is to copy the data from the ThreadLocal event object to the RingBuffer.

S306: Publish an event.

Use the publish(sequence) method of RingBuffer to publish events and notify consumers that there are new events to process. This is also a lock-free operation, and thread safety is guaranteed by the CAS mechanism.

S4. Event processing.

S401. Get events from RingBuffer.

The consumer thread gets events from the RingBuffer and uses the sequence number of the consumer thread to identify the events to be processed. RingBuffer ensures that events are read by consumers in the order in which they are published by the producer.

S402: Obtain thread local processing status.

Get the local processing state object of the consumer thread through ThreadLocal (such as

`ProcessingState`). Each consumer thread has an independent processing state object, avoiding data sharing and competition between threads.

S403: Process transaction data.

Use the thread-local processing state object to process the transaction data obtained from the RingBuffer. The processing logic can be customized according to specific needs, such as calculation, storage, filtering and other operations.

S404: Update processing status.

After processing the transaction data, update the thread-local processing state object. This ensures the independence and thread safety of the processing state.

S405: Execute subsequent operations.

After processing the transaction data, some subsequent operations will be performed based on the processing results, such as logging, notifying other systems, etc. These operations are also completed locally to ensure that they do not affect the execution of other threads.

This embodiment uses the CAS mechanism in multi-thread synchronization to ensure thread safety. RingBuffer uses the CAS (Compare-And-Swap) mechanism to achieve lock-free sequence number acquisition and event publishing, ensuring thread safety between producers and consumers. This mechanism reduces lock contention and improves system performance.

The design of RingBuffer ensures the orderly processing of events. Consumers process events in the order published by producers, ensuring data consistency and processing accuracy.

RingBuffer supports multi-producer and multi-consumer modes. Through an efficient synchronization mechanism, multiple producers can publish events at the same time, and multiple consumers can process events at the same time, adapting to the needs of high-concurrency environments. Through lock-free design and pre-allocated memory, RingBuffer avoids data consistency issues. Event objects are pre-allocated in RingBuffer to ensure data consistency and processing accuracy in a high-concurrency environment. By reducing lock contention and context switching, using CAS mechanism and ThreadLocal storage, the present invention significantly optimizes system performance and improves the throughput and response speed of financial transaction systems.

In the financial transaction scenario, this embodiment combines RingBuffer and ThreadLocal to improve the single-machine processing performance, which has the following significant advantages:

High throughput: Lock-free design: RingBuffer adopts lock-free design, through CAS

The (Compare-And-Swap) mechanism realizes data production and consumption, reduces lock contention and context switching, and improves system throughput. Pre-allocated memory: Event objects are pre-allocated in the RingBuffer, avoiding frequent memory allocation and garbage collection at runtime, and improving the overall performance of the system.

Low latency: Thread local storage: Use ThreadLocal to provide independent storage space for each thread, avoiding data sharing and competition between threads, significantly reducing the latency of data processing. Efficient waiting strategy: By selecting an appropriate waiting strategy (such as BusySpinWaitStrategy or YieldingWaitStrategy), the latency of event processing is further reduced, meeting the strict requirements of financial trading systems for low latency.

Thread safety: Independent thread local variables: ThreadLocal variables ensure that each thread has an independent event object and processing status, avoiding data competition between threads and improving the thread safety of the system. Ordered event processing: RingBuffer ensures that events are read and processed by consumers in the order published by producers, ensuring data consistency and processing accuracy.

Easy scalability: Support multiple producers and multiple consumers: RingBuffer supports multiple producers and multiple consumers. Through an efficient synchronization mechanism, multiple producers can publish events at the same time, and multiple consumers can process events at the same time, adapting to the needs of transaction systems of different scales.

Flexible configuration: According to the specific needs of the system, the size and waiting strategy of RingBuffer can be flexibly configured to meet different load and performance requirements.

Efficient resource utilization: Reduce context switching: Lock-free design and efficient waiting strategy reduce context switching between threads, reduce system overhead, and improve CPU resource utilization.

Optimize memory usage: ThreadLocal and pre-allocated memory mechanisms reduce the frequency of garbage collection, optimize memory usage, and improve system stability and performance.

Strong adaptability: Coping with high-concurrency scenarios: The present invention is particularly suitable for high-concurrency data processing scenarios in financial transaction systems, and can efficiently process a large number of concurrent transaction requests to ensure high availability and reliability of the system.

Wide application: In addition to financial transaction scenarios, the method and system of the present invention are also applicable to other fields that require high-performance data processing, such as real-time data analysis, online advertising bidding, Internet of Things data processing, etc.

By combining RingBuffer and ThreadLocal, this embodiment provides an efficient, low-latency, and thread-safe financial transaction system data processing method. It significantly improves the single-machine processing performance, meets the strict requirements of high concurrency, high throughput, and low latency, and has broad application prospects and market value.

In addition, it should be noted that: an embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the method described in the above embodiment is executed.

In addition, it should be noted that an embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor loads and executes the computer program, the method described in the above embodiment is executed.

The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims (10)

1. A data processing method for a financial transaction scenario, characterized in that the method comprises: Initialize the ring buffer; Use ThreadLocal for thread local storage; In the producer thread, obtain the thread local event object through ThreadLocal and write the transaction data into the thread local event object; Get the next available sequence number in the ring buffer, and use the obtained sequence number to get the corresponding event object from the ring buffer; Write the data in the thread local event object to the event object in the ring buffer; Publish events through the ring buffer to notify consumers that new events are available for processing; The consumer thread obtains events from the ring buffer, uses the serial number of the consumer thread to identify the events to be processed, and obtains the local processing status object of the consumer thread through ThreadLocal; Use the thread-local processing state object to process the transaction data from the ring buffer; After processing the transaction data, update the thread-local processing status object. 2. A data processing method for a financial transaction scenario according to claim 1, characterized in that the initializing the ring buffer comprises: Select the buffer based on the expected concurrency and data processing requirements of the trading system; Choose a wait strategy to balance system latency and CPU usage; Create an event factory, which is used to create event objects in the ring buffer; Initialize the Disruptor framework, create and manage the ring buffer through the Disruptor framework; Configure the event handler, which is used to process events read from the ring buffer. 3. A data processing method for a financial transaction scenario according to claim 1, characterized in that the use of ThreadLocal for thread local storage comprises: Determine the objects that need ThreadLocal storage; Create ThreadLocal variables for each object that requires thread local storage; Initialize ThreadLocal variables. During system initialization, initialize ThreadLocal variables for each thread. Get the ThreadLocal object in the thread. When processing transaction data, get the object stored locally in the thread through the ThreadLocal variable. In the producer thread, write the transaction data into the transaction event object stored in ThreadLocal; In the consumer thread, the processing status object is obtained through the ThreadLocal variable. 4. A data processing method for a financial transaction scenario as described in claim 3, characterized in that the stored objects include transaction event objects and processing status objects, the transaction event objects include transaction ID and transaction price, and the processing status objects include intermediate states of transaction processing and results of transaction processing. 5. A data processing method for a financial transaction scenario as described in claim 1, characterized in that after processing the transaction data, a log recording operation is performed or other systems are notified based on the processing results. 6. A data processing method for a financial transaction scenario as described in claim 1, characterized in that the ring buffer implements lock-free serial number acquisition and event publishing through a Compare-And-Swap mechanism. 7. A data processing method for financial transaction scenarios as described in claim 1, characterized in that the ring buffer supports multi-producer and multi-consumer modes, and through a synchronization mechanism, multiple producers can publish events at the same time, and multiple consumers can process events at the same time. 8. A data processing method for financial transaction scenarios as described in claim 1, characterized in that through lock-free design and pre-allocated memory, event objects are pre-allocated in a ring buffer to ensure data consistency and processing accuracy in a high concurrency environment. 9. A computer-readable storage medium storing a computer program, wherein when the computer program is loaded and executed by a processor, the method according to any one of claims 1 to 8 is adopted. 10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor adopts the method according to any one of claims 1 to 8 when loading and executing the computer program.