Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/46—Multiprogramming arrangements
  • G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
  • G06F9/526—Mutual exclusion algorithms
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
  • G06F12/02—Addressing or allocation; Relocation
  • G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
  • G06F12/0284—Multiple user address space allocation, e.g. using different base addresses
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
  • G06F12/02—Addressing or allocation; Relocation
  • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
  • G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
  • G06F12/0806—Multiuser, multiprocessor or multiprocessing cache systems
  • G06F12/0815—Cache consistency protocols
  • G06F12/0817—Cache consistency protocols using directory methods
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F8/00—Arrangements for software engineering
  • G06F8/40—Transformation of program code
  • G06F8/41—Compilation
  • G06F8/45—Exploiting coarse grain parallelism in compilation, i.e. parallelism between groups of instructions
  • G06F8/457—Communication
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/46—Multiprogramming arrangements
  • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
  • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
  • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
  • G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/46—Multiprogramming arrangements
  • G06F9/54—Interprogram communication
  • G06F9/544—Buffers; Shared memory; Pipes
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
  • G06F12/02—Addressing or allocation; Relocation
  • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
  • G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
  • G06F12/0806—Multiuser, multiprocessor or multiprocessing cache systems
  • G06F12/0815—Cache consistency protocols
  • G06F12/0837—Cache consistency protocols with software control, e.g. non-cacheable data
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F2209/00—Indexing scheme relating to G06F9/00
  • G06F2209/52—Indexing scheme relating to G06F9/52
  • G06F2209/523—Mode

Definitions

• the invention relates generally to the field of computing systems in which multiple processors share memory but in which each is provided with separate access to input-output (I/O) devices such as disks. More particularly, the invention relates to computer science techniques that utilize a shared memory disk (MEMDISK).
• MEMDISK shared memory disk
• the clustering of workstations is a well-known art. In the most common cases, the clustering involves workstations that operate almost totally independently, utilizing the network only to share such services as a printer, license-limited applications, or shared files.
• some software packages allow a cluster of workstations to share work.
• the work arrives, typically as batch jobs, at an entry point to the cluster where it is queued and dispatched to the workstations on the basis of load.
• the operating system and cluster subsystem are built around the concept of message-passing.
• message-passing means that a given workstation operates on some portion of a job until communications (to send or receive data, typically) with another workstation is necessary. Then, the first workstation prepares and communicates with the other workstation.
• MPP Massively Parallel Processor
• the passing of a message is an extremely expensive operation; expensive in the sense that many CPU cycles in the sender and receiver are consumed by the process of sending, receiving, bracketing, verifying, and routing the message, CPU cycles that are therefore not available for other operations.
• a highly streamlined message-passing subsystem can typically require 10,000 to 20,000 CPU cycles or more.
• Message-passing parallel processor systems have been offered commercially for years but have failed to capture significant market share because of poor performance and difficulty of programming for typical parallel applications. Message-passing parallel processor systems do have some advantages. In particular, because they share no resources, message-passing parallel processor systems are easier to provide with high-availability features. What is needed is a better approach to parallel processor systems. There are alternatives to the passing of messages for closely-coupled cluster work. One such alternative is the use of shared memory for inter- processor communication.
• the processors shared a single copy of the operating system.
• the problem with such systems is that they cannot be efficiently scaled beyond four to eight way systems except in unusual circumstances. All known cases of said unusual circumstances are such that the systems are not good price-performance systems for general-purpose computing.
• U.S. Patent Applications 09/273,430, filed March 19, 1999 and PCT/US00/01262, filed January 18, 2000 are hereby expressly incorporated by reference herein for all purposes.
• U.S. Ser. No. 09/273,430 improved upon the concept of shared memory by teaching the concept which will herein be referred to as a tight cluster.
• the concept of a tight cluster is that of individual computers, each with its own CPU(s), memory, I/O, and operating system, but for which collection of computers there is a portion of memory which is shared by all the computers and via which they can exchange information.
• 09/273,430 describes a system in which each processing node is provided with its own private copy of an operating system and in which the connection to shared memory is via a standard bus.
• the advantage of a tight cluster in comparison to an SMP is "scalability" which means that a much larger number of computers can be attached together via a tight cluster than an SMP with little loss of processing efficiency. What is needed are improvements to the concept of the tight cluster.
• PCs personal computers
• workstations use caches except for the very low end of the PC business where caches are omitted for price reasons and performance is, therefore, poor.
• Caches however, present a problem for shared-memory computing systems; the problem of coherence.
• a particular processor reads or writes a word of shared memory
• That word and usually a number of surrounding words are transferred to that particular processor's cache memory transparently by cache-memory hardware. That word and the surrounding words (if any) are transferred into a portion of the particular processor's cache memory that is called a cache line or cache block.
• the representation in the cache memory will become different from the value in shared memory.
• That cache line within that particular processor's cache memory is, at that point, called a "dirty" line.
• the particular processor with the dirty line when accessing that memory address will see the new (modified) value.
• Other processors accessing that memory address will see the old (unmodified) value in shared memory. This lack of coherence between such accesses will lead to incorrect results.
• Modern computers, workstations, and PCs which provide for multiple processors and shared memory, therefore, also provide high-speed, transparent cache coherence hardware to assure that if a line in one cache changes and another processor subsequently accesses a value which is in that address range, the new values will be transferred back to memory or at least to the requesting processor.
• Caches can be maintained coherent by software provided that sufficient cache-management instructions are provided by the manufacturer. However, in many cases, an adequate arsenal of such instructions are not provided. Moreover, even in cases where the instruction set is adequate, the software overhead is so great that no examples of are known of commercially successful machines which use software-managed coherence. SUMMARY OF THE INVENTION A goal of the invention is to simultaneously satisfy the above-discussed requirements of improving and expanding the tight cluster concept which, in the case of the prior art, are not satisfied.
• One embodiment of the invention is based on a method, comprising: setting aside a particular range of a shared memory as a MEMDISK; and providing control for each of several operating systems that compose processing nodes coupled to said shared memory such that none of the processing nodes will attempt to utilize pages within said particular region for non-MEMDISK purposes.
• Another embodiment of the invention is based on a system, comprising a multiplicity of processors, each with some private memory and the multiplicity with some shared memory, interconnected and arranged such that memory accesses to a first set of address ranges will be to local, private memory whereas memory accesses to a second set of address ranges will be to shared memory, and arranged such that at least some of said processors are provided with input-output subsystems and that said input-output (I/O) traffic started by one processor for an I/O device attached to another processor will be started by inter-processor signals but continued via use of a portion of shared memory accessed via I/O driver emulation means.
• I/O input-output
• Another embodiment of the invention is based on a computer system which provides operating system extensions to perform disk input-output (I/O) functions in a shared-memory environment, where said extensions perform the functions with direct Load and Store operations.
• Another embodiment of the invention is based on a computer system that provides system-wide registration of shared-memory disk partitions at all of a multiplicity of processing nodes within the system.
• Another embodiment of the invention is based on a computer system that provides system-wide registration of shared-memory disk access methodologies at all of a multiplicity of processing nodes within the system.
• Another embodiment of the invention is based on a computer system that provides system-wide status of shared-memory disk operations at all of a multiplicity of processing nodes within the system.
• FIG. 1 illustrates a block schematic view of a system, representing an embodiment of the invention.
• FIG. 2 illustrates a block schematic view of another system, representing an embodiment of the invention.
• DESCRIPTION OF PREFERRED EMBODIMENTS The invention and the various features and advantageous details thereof are explained more fully with reference to the nonlimiting embodiments that are illustrated in the accompanying drawings and detailed in the following description of preferred embodiments. Descriptions of well known components and processing techniques are omitted so as not to unnecessarily obscure the invention in detail.
• U.S. Ser. No. 09/273,430 include a system which is a single entity; one large supercomputer.
• the invention is also applicable to a cluster of workstations, or even a network.
• the invention is applicable to systems of the type of Pfister or the type of U.S. Ser. No. 09/273,430 in which each processing node has its own copy of an operating system.
• the invention is also applicable to other types of multiple processing node systems.
• the context of the invention can include a tight cluster as described in
• a tight cluster is defined as a cluster of workstations or an arrangement within a single, multiple-processor machine in which the processors are connected by a high-speed, low-latency interconnection, and in which some but not all memory is shared among the processors.
• accesses to a first set of ranges of memory addresses will be to local, private memory but accesses to a second set of memory address ranges will be to shared memory.
• the significant advantage to a tight cluster in comparison to a message-passing cluster is that, assuming the environment has been appropriately established, the exchange of information involves a single STORE instruction by the sending processor and a subsequent single LOAD instruction by the receiving processor.
• WSGR 15245-712; WSGR 15245-713; WSGR 15245-715; WSGR 15245-716; WSGR 15245-717; WSGR 15245-719; and WSGR 15245-720 can be performed in such a way as to require relatively little system overhead, and to be done once for many, many information exchanges. Therefore, a comparison of 10,000 instructions for message-passing to a pair of instructions for tight-clustering, is valid.
• the invention can include providing highly-efficient operating system control for a tight cluster system.
• a shared memory disk MEMDISK which can be shared among the various processes and processors of the cluster.
• the invention can include the utilization of shared memory to achieve OS-transparent high-speed access to disk.
• the invention can also provide the ability to substitute memory accesses for disk accesses under certain circumstances, while maintaining OS -transparency to the substitution.
• the invention is applicable to the kind of systems taught by U.S. Ser. No. 09/273,430 and is also applicable to other architectures such as NUMA, CC-NUMA, and other machines in which each processor or processor aggregation is provided with separate I/O.
• NUMA NUMA
• CC-NUMA CC-NUMA
• NUMA machines all of memory is shared and there is one copy of the operating system.
• U.S. Ser. No. 09/273,430 only a portion of the memory is shared and each node is provided with a separate copy of the operating system.
• the paths from each of the processing nodes to non-local I/O is generally one of several types. These types include: (1) each of the separate processing nodes restricted from reaching I/O attached to other nodes; (2) each of the processing nodes acting as a surrogate for another node, providing I/O capability to the requesting node in a proxy fashion; (3) in schemes such as
• NUMA and CC-NUMA where a processing node may be provided with a path, via directory and cache-coherence means, whereby the single common operating system can reach I/O on the other nodes; or (4) a system external provision providing a common I/O resource, such as Fibre Channel I/O, a twin- tailed SCSI disk facility, or standard networking facility.
• the present invention can be used in the context of the environment described in U.S. Ser. No. 09/273,430 where multiple computers are provided with means to selectively address a first set of memory address ranges which will be to private memory and a second set of memory ranges which will be to shared memory.
• the invention can include: setting aside a particular range of shared memory as a MEMDISK; providing control means for each of the several operating systems such that none will attempt to utilize pages within this region for normal shared-memory purposes.
• a device driver interface can be utilized that is responsive to disk I/O operations to a particular disk and which will translate said I/O operations to LOAD and STORE operations to said region of memory. I/O operations can be intercepted which may otherwise be directed to a physical disk and redirect those operations to said region of memory.
• each processor is provided with its own memory and its own operating system or micro-kernel and may be provided with its own I/O subsystem.
• the present invention is applicable in such a system where more than one such processor is provided with I/O and in which the file system is visible to all or a multiplicity of processors.
• a particular processor When a particular processor develops traffic for an I/O device not on its I/O subsystem, it requests that the processor owning the particular I/O subsystem satisfy the request. In this way, the processors serve as I/O processors to each other.
• the requesting process sends to the service process a request sufficient to initiate and completely satisfy the particular I/O block request.
• the movement is accomplished via the shared MEMDISK.
• the MEMDISK as described here, teaches three major new concepts.
• the purpose is not to minimize access time, although that minimization occurs, but rather is to minimize interference between a pair of processors so that a first process on a first processor may deliver data to the MEMDISK as it becomes available from the file system and a second process on a second processor may read the data from the shared memory region without interrupting the first processor after the initial START I/O process is begun.
• MEMDISK WRITES are protected by semaphores to eliminate data corruption, and MEMDISK semaphores are applied on a region-by-region basis so that artificial interference is also eliminated.
• the second major concept is that of disk-to-memory dynamic redirection.
• the initial START I/O to a particular file is delivered, by the device driver, to the processor owning the I/O device.
• Subsequent I/O to that file is dynamically redirected to the MEMDISK by the device driver and is managed transparent to all other processes including the remainder of the operating system.
• MEMDISK disk-to-memory dynamic redirection
• a third major teaching of this invention is that of "aging" of data within the MEMDISK region. Once I/O data is placed in the region, it stays there and is available to any process on any processor until the MEMDISK region becomes full, at which time older (least-recently-used) information is replaced by newer information. The portion of the MEMDISK reflecting data on a particular I/O device is kept coherent by the owner of the I/O device.
• This invention describes a means outside the operating system but within shared memory to provide any node in a shared-memory computing system access to memory, which is physically attached to another node.
• each system is provided with some local, private memory and a separate copy of the operating system.
• each of several nodes is provided with its own I/O channel to disks and other I/O units.
• the operating system in each node is augmented with external extensions, not part of the operating system, which provide means by which said extensions have capabilities to reach shared memory and to communicate to other said nodes via Load and Store instructions to shared memory.
• the invention includes other extensions called shared-memory-disk (SMD) extensions, which make use of primitives and by which disk I/O functions which originate in applications and which are then passed to the operating system are processed.
• Said disk I/O functions, arriving at the Operating System, are processed by said SMD extensions and are translated to shared-memory Load and Store instructions. This effects disk I/O transactions issued by the Operating System into Load and Store transfers via shared memory, and so satisfies the Operating System I/O request transparently.
• SMD shared-memory-disk
• An additional key teaching of this invention is the system-wide registration of all shared-memory disk partitions at all processing nodes within the system, with an access methodology dependent upon performance requirements.
• additional key teaching of this invention is the use of a means herein called "software RAID" to assure high-availability of the disk I/O portion of the computing system.
• An additional key feature of this system is the provision, in shared memory, for retention of data which passes through shared memory so that subsequent accesses to said retained images can be satisfied solely by memory transfers thus achieving dramatic performance improvements.
• Each of the computing nodes that participates in constructing an SMD instantiation contributes a node-local data store to the instantiation hereafter referred to as a SmdBlock.
• the data store is not limited as to type, and may be private memory, paged virtual memory, local disk I/O, etc.
• a preferred implementation provides this data store with commodity disk drives.
• These SmdBlocks are collected and logically bound into a SmdDisk in a repeatable order. This order is subsequently used to base an access methodology most appropriate to use (e.g., ordinally-arranged contiguity for best random access, interleaved contiguity for best sequential access, distributed contiguity for de- clustered failover or software Raid applications, etc.).
• the SmdDisk and SmdBlocks are identified within the immediate community of computing nodes by a community-unique signature.
• a known address (or known key to address translation) is searched for a single data structure (hereafter referred to as the SmdAnchor) containing identifying patterns, validity and version information, and a shared-memory pointer to a linked list of SmdDisk control structures. If the SmdAnchor is not yet present, one is created. If the SmdAnchor is present, the SmdDisk list is searched for a matching signature, which informs the instantiation that the supporting shared-memory data structures have been created by another computing node, and it may now access same as appropriate.
• SmdDisk structure If the SmdDisk structure is not present, it is created and linked into the SmdAnchor list, for future SmdDisk and SmdBlock instantiations to find.
• the passive nature of the preferred implementation avoids inter-node messaging and communications overhead, while simplifying state transitions.
• the SmdAnchor Upon instantiation of a SmdBlock, the SmdAnchor, and its corresponding SmdDisk list, is searched for a matching signature. If one is not found, the search is repeated ad infinitum, with an appropriate delay to allow for another computing node or computing process to create the SmdDisk structure. If and when one is found, the SmdBlock inserts its computing node-unique information into the SmdDisk structure. When all SmdBlocks that contribute to a SmdDisk have inserted their information into the control structure, the SmdDisk is ready for use, and thusly describes a distributed data store.
• the SmdDisk structure can be examined and the operation broken into a list of transactions matching the SmdBlock node-locality and requirements of the SmdDisk data store access methodology. These transactions are then sent to each of the SmdBlock nodes for fulfillment using shared-memory Load and Store operations to transfer the data between nodes.
• the SmdDisk allocates shared- memory data areas for each of the transactions and sends read commands to the SmdBlock nodes referencing said data areas.
• the SmdBlock nodes perform the node-local data store reads, moving the data into said areas, then return status to the original node.
• the originating node collects all SmdBlock transaction statuses (either passively or via the mechanisms described in [2]), the operation is considered complete and the data contained in the shared-memory data areas can be used to fulfill the original request.
• the SmdDisk allocates shared-memory data areas for each of the transactions and moves the write data into said areas, according to the access methodology, then sends write commands to the appropriate SmdBlock nodes.
• the SmdBlock nodes perform the data store writes, moving the data from said areas into their data store, then return status to the originating node.
• the originating node collects all SmdBlock transaction statuses, the operation is complete.
• the shared-memory areas used to transfer the data can be kept resident in shared memory, and subsequently used to satisfy read requests by any node connected to the shared-memory system.
• the management of a shared-memory cache requires the use of a shared-memory mutual exclusion mechanisms to maintain coherency. This shared nature allows multiple nodes to access, and contribute to, the shared cache, resulting in being able to completely satisfy Disk I/O operations with Load and Store operations.
• control structures to manage said shared-memory cache can be kept within the SmdDisk control structure, allowing the above mentioned search and access methods to be used.
• This cache can result in a significant performance enhancements, as the latency and transfer time for a node to deliver data into shared memory, as well as the access of the physical data store, is eliminated.
• An extension of the invention allows different access methodologies to be implemented for differing requirements, without affecting the Operating System perceived implementation.
• the data stores contributed by all SmdBlock nodes can be ordinally arranged as contiguous data stores using commodity disk drives. This allows the head movement latency to be mitigated across the SmdBlock contributors, providing greatly reduced access latency and improved performance.
• the data stores can be arranged in a striping, or RaidO configuration, thus improving throughput by effecting concurrent media access.
• the SmdBlocks can implement a "software RAID” by striping in a Chained Declustering methodology allowing head movement mitigation and concurrent access (at the expense of duplicate data store space).
• the SmdBlocks can implement another "software RAID” by striping the SmdBlock nodes' contributions in a RAID5 methodology, balancing the computing cost of data parity generation and checking with the improvements provided by head movement mitigation and concurrent access.
• the shared-memory caching is used in conjunction with a "software RAID" methodology, for a complete high performance fault tolerant implementation. The Operating System perceived implementation of the data store remains, in all cases, transparent.
• preferred embodiments of the invention can be identified one at a time by testing for the substantially highest performance.
• the test for the substantially highest performance can be carried out without undue experimentation by the use of a simple and conventional benchmark (speed) experiment.
• substantially is defined as at least approaching a given state (e.g., preferably within 10% of, more preferably within 1% of, and most preferably within 0.1% of).
• coupled is defined as connected, although not necessarily directly, and not necessarily mechanically.
• the term means, as used herein, is defined as hardware, firmware and/or software for achieving a result.
• program or phrase computer program as used herein, is defined as a sequence of instructions designed for execution on a computer system.
• a program may include a subroutine, a function, a procedure, an object method, an object implementation, an executable application, an applet, a servlet, a source code, an object code, and/or other sequence of instructions designed for execution on a computer system.
• the invention is useful in conjunction with data input and transformation (such as are used for the purpose of speech recognition), or in conjunction with transforming the appearance of a display (such as are used for the purpose of video games), or the like.
• data input and transformation such as are used for the purpose of speech recognition
• transforming the appearance of a display such as are used for the purpose of video games
• a system representing an embodiment of the invention, can be cost effective and advantageous for at least the following reasons.
• the invention improves the speed of parallel computing systems.
• the invention improves the scalability of parallel computing systems.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Software Systems (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Multi Processors (AREA)
• Memory System Of A Hierarchy Structure (AREA)
• Hardware Redundancy (AREA)
• Information Transfer Systems (AREA)

Applications Claiming Priority (7)

Publications (1)

Family

ID=27387201

Family Applications (3)

Family Applications After (2)

Country Status (4)

Families Citing this family (13)

Family Cites Families (31)

• 2000
  • 2000-08-31 EP EP00959876A patent/EP1214653A2/de not_active Withdrawn
  • 2000-08-31 AU AU71007/00A patent/AU7100700A/en not_active Abandoned
  • 2000-08-31 WO PCT/US2000/024039 patent/WO2001016760A1/en not_active Ceased
  • 2000-08-31 CA CA002382929A patent/CA2382929A1/en not_active Abandoned
  • 2000-08-31 AU AU71083/00A patent/AU7108300A/en not_active Abandoned
  • 2000-08-31 EP EP00957948A patent/EP1214651A2/de not_active Withdrawn
  • 2000-08-31 CA CA002382728A patent/CA2382728A1/en not_active Abandoned
  • 2000-08-31 EP EP00959824A patent/EP1214652A2/de not_active Withdrawn
  • 2000-08-31 AU AU74742/00A patent/AU7474200A/en not_active Abandoned
  • 2000-08-31 AU AU69496/00A patent/AU6949600A/en not_active Abandoned
  • 2000-08-31 AU AU71121/00A patent/AU7112100A/en not_active Abandoned
  • 2000-08-31 AU AU69497/00A patent/AU6949700A/en not_active Abandoned
  • 2000-08-31 WO PCT/US2000/024248 patent/WO2001016742A2/en not_active Ceased
  • 2000-08-31 AU AU71100/00A patent/AU7110000A/en not_active Abandoned
  • 2000-08-31 CA CA002382927A patent/CA2382927A1/en not_active Abandoned
  • 2000-08-31 WO PCT/US2000/024217 patent/WO2001016741A2/en not_active Ceased
  • 2000-08-31 WO PCT/US2000/024216 patent/WO2001016761A2/en not_active Ceased
  • 2000-08-31 AU AU71085/00A patent/AU7108500A/en not_active Abandoned
  • 2000-08-31 WO PCT/US2000/024210 patent/WO2001016740A2/en not_active Ceased
  • 2000-08-31 WO PCT/US2000/024329 patent/WO2001016750A2/en not_active Ceased
  • 2000-08-31 WO PCT/US2000/024150 patent/WO2001016738A2/en not_active Ceased
  • 2000-08-31 WO PCT/US2000/024298 patent/WO2001016743A2/en not_active Ceased
  • 2000-08-31 AU AU71136/00A patent/AU7113600A/en not_active Abandoned
  • 2000-08-31 WO PCT/US2000/024147 patent/WO2001016737A2/en not_active Ceased

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events