WO2012121063A1 - 翻字処理装置、プログラム、記録媒体、及び方法 - Google Patents
翻字処理装置、プログラム、記録媒体、及び方法 Download PDFInfo
- Publication number
- WO2012121063A1 WO2012121063A1 PCT/JP2012/054956 JP2012054956W WO2012121063A1 WO 2012121063 A1 WO2012121063 A1 WO 2012121063A1 JP 2012054956 W JP2012054956 W JP 2012054956W WO 2012121063 A1 WO2012121063 A1 WO 2012121063A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- transliteration
- probability
- character string
- rewrite
- spelling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/40—Processing or translation of natural language
- G06F40/42—Data-driven translation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/126—Character encoding
- G06F40/129—Handling non-Latin characters, e.g. kana-to-kanji conversion
Definitions
- the present invention relates to a transliteration processing device, a transliteration processing program, a computer-readable recording medium that records the transliteration processing program, and a transliteration processing method.
- Non-Patent Document 1 a transliteration model transliterated string s n of a language into a character string t n in other languages, names represented by the string s n of the one language Chinese, Japanese, and English
- a method of performing transliteration using a switched model according to which one of the two is the origin for example, Non-Patent Document 1.
- Non-Patent Document 2 a character string or a character (hereinafter referred to as a fragment) ⁇ constituting a character string s n of a certain language is rewritten to a fragment ⁇ constituting a character string t n of another language.
- a ⁇ method for calculating a transliteration probability that a character string s n in a certain language is transliterated into a character string t n in another language for example, Non-Patent Document 2.
- the present invention has been made in view of such a point, and the purpose of the present invention is to transliterate even if it is not clear what word the character string to be transliterated originates from.
- Transliteration processing apparatus, transliteration processing program, computer-readable recording medium recording transliteration processing program, and transliteration processing method capable of specifying whether character string is transliteration of target concerned with higher accuracy than before Is to provide.
- a transliteration processing apparatus provides: From the training set including a plurality of transliteration pairs of an original spelling character string composed of a spelling in an arbitrary origin language and a target spelling character string composed of a spelling obtained by translating the original spelling character string into a predetermined target language, Different from each other, including a plurality of sets of an original fragment constituting a spelling character string, a transliteration fragment constituting the target spelling character string, and a rewriting probability that the original fragment is rewritten into the transliteration fragment for transliteration.
- a generating unit that generates K rewrite tables corresponding to K origin languages and K transliteration tables corresponding to the K origin languages including a plurality of transliteration pairs included in the training set; For each transliteration pair included in the training set, the original spelling character string of the transliteration pair corresponds to the rewrite table by using each of the rewrite probabilities included in the K rewrite tables.
- the origin language is the origin
- the transliteration probability of translating to the target spelling string of the transliteration pair is calculated, and then the transliteration probability is associated with the transliteration pair to correspond to the origin language.
- An update unit that repeats the calculation of the transliteration probability and the update of the rewrite probability after updating the rewrite probability included in each of the K rewrite tables so as to maximize the value calculated in It is characterized by providing.
- each of the original spelling character strings originating in the origin language corresponding to each of the K transliteration tables indicates a ratio of a plurality of original spelling character strings included in the training set.
- An initialization unit for initializing the weight parameter for each of a plurality of transliteration pairs included in the training set, based on the weighted average of the transliteration probabilities included in the K transliteration tables by the weight parameter, the original spelling
- the update unit for each of a plurality of transliteration pairs included in the training set, based on the weighted average of the transliteration probabilities included in the K transliteration tables by the weight parameter, the original spelling
- the generating unit generates the K rewriting tables by changing the rewriting probability included in the generated one rewriting table for each of the K languages after generating the one rewriting table.
- Corresponding K rewrite tables may be generated.
- the generation unit generates one transliteration table by an ⁇ method using one rewrite table, and then changes a transliteration probability included in the generated one transliteration table.
- Generate K transliteration tables corresponding to the K languages The updating unit may update transliteration probabilities included in each of the K transliteration tables by an EM algorithm.
- the update unit may repeat the update until an amount of change in an expected value of the likelihood function due to the update becomes less than a predetermined magnitude.
- a receiving unit that receives a first character string composed of spellings in the arbitrary origin language and a second character string composed of spellings in the predetermined target language; Using each of the K transliteration tables, a transliteration probability that the accepted first character string is transliterated to the accepted second character string is obtained, and the obtained transliteration probability is output.
- a probability output unit May be further provided.
- the reception unit may receive a first character string that is not included in the training set.
- a transliteration processing program provides: Computer From the training set including a plurality of transliteration pairs of an original spelling character string composed of a spelling in an arbitrary origin language and a target spelling character string composed of a spelling obtained by translating the original spelling character string into a predetermined target language, Different from each other, including a plurality of sets of an original fragment constituting a spelling character string, a transliteration fragment constituting the target spelling character string, and a rewriting probability that the original fragment is rewritten into the transliteration fragment for transliteration.
- a generating unit that generates K rewrite tables corresponding to K origin languages and K transliteration tables corresponding to the K origin languages including a plurality of transliteration pairs included in the training set; For each transliteration pair included in the training set, the original spelling character string of the transliteration pair corresponds to the rewrite table by using each of the rewrite probabilities included in the K rewrite tables.
- the origin language is the origin
- the transliteration probability of translating to the target spelling string of the transliteration pair is calculated, and then the transliteration probability is associated with the transliteration pair to correspond to the origin language.
- An expected value of a likelihood function for calculating the likelihood representing the likelihood of the K transliteration tables when the training set is obtained after saving in the transliteration table, and using the transliteration probability An update unit that repeats the calculation of the transliteration probability and the update of the rewrite probability after updating the rewrite probability included in each of the K rewrite tables so as to maximize the value calculated in It is made to function as.
- a computer-readable recording medium provides: Computer From the training set including a plurality of transliteration pairs of an original spelling character string composed of a spelling in an arbitrary origin language and a target spelling character string composed of a spelling obtained by translating the original spelling character string into a predetermined target language, Different from each other, including a plurality of sets of an original fragment constituting a spelling character string, a transliteration fragment constituting the target spelling character string, and a rewriting probability that the original fragment is rewritten into the transliteration fragment for transliteration.
- a generating unit that generates K rewrite tables corresponding to K origin languages and K transliteration tables corresponding to the K origin languages including a plurality of transliteration pairs included in the training set; For each transliteration pair included in the training set, the original spelling character string of the transliteration pair corresponds to the rewrite table by using each of the rewrite probabilities included in the K rewrite tables.
- the origin language is the origin
- the transliteration probability of translating to the target spelling string of the transliteration pair is calculated, and then the transliteration probability is associated with the transliteration pair to correspond to the origin language.
- An expected value of a likelihood function for calculating the likelihood representing the likelihood of the K transliteration tables when the training set is obtained after saving in the transliteration table, and using the transliteration probability An update unit that repeats the calculation of the transliteration probability and the update of the rewrite probability after updating the rewrite probability included in each of the K rewrite tables so as to maximize the value calculated in The transliteration processing program characterized by making it function as is recorded.
- the transliteration processing method includes: A transliteration processing device including a generation unit and an update unit, Training in which the generation unit includes a plurality of transliteration pairs of an original spelling character string composed of spellings in an arbitrary origin language and a target spelling character string composed of a spelling obtained by translating the original spelling character string into a predetermined target language. From the set, a set of an original fragment constituting the original spelling character string, a transliteration fragment constituting the target spelling character string, and a rewrite probability that the original fragment is rewritten to the transliteration fragment for transliteration.
- Generation steps to generate The update unit uses each rewrite probability included in the K rewrite tables for each transliteration pair included in the training set, so that the original spelling character string of the transliteration pair is When the origin language corresponding to the rewrite table is the origin, calculate the transliteration probability to be translated into the target spelling string of the transliteration pair, and then associate the transliteration probability with the transliteration pair An expected value of a likelihood function for calculating a likelihood representing the likelihood of the K transliteration tables when the training set is obtained after storing in the transliteration table corresponding to the origin language, After updating the rewrite probability included in each of the K rewrite tables so as to maximize the value calculated using the transliteration probability, the calculation of the transliteration probability and the update of the rewrite probability are repeated. Update steps, It is characterized by having.
- the number of words in the character string to be transliterated is Even if it is unknown whether it is the origin, it can be specified with higher accuracy than before whether the transliterated character string is a transliteration of the target.
- the communication network 10 includes a computer communication network 10 (hereinafter simply referred to as the communication network 10), a transliteration processing device 100 according to the present invention, a terminal device 200, and a Japanese database server (hereinafter referred to as a Japanese DB). Server) 301 and a foreign language database server (hereinafter referred to as a foreign language DB server) 302.
- the communication network 10 is configured by the Internet, for example.
- the communication network 10 may be a LAN (Local Area Network) or a public line network.
- the terminal apparatus 200 Before describing the transliteration processing apparatus 100, the terminal apparatus 200, the Japanese DB server 301, and the foreign language DB server 302 will be described.
- the terminal device 200 is composed of a personal computer including a display unit such as an LCD (Liquid Crystal Display) and an input unit such as a keyboard.
- the terminal device 200 displays an input screen FI1 as shown in FIG. 2A so as to prompt the user to input an input pair.
- This input pair is a character string that the user thinks is a first character string composed of spellings in an arbitrary origin language and a character string obtained by translating the first character string into a character string of a predetermined target language, A pair with a second character string composed of spellings in a predetermined target language.
- Transliteration refers to phonological translation between languages with different notation systems, such as translating “Barak Obama” to “Barak Obama”.
- the predetermined target language is Japanese
- the arbitrary source language is, for example, a language that uses alphabets for notation such as English, French, Spanish, German, Polish, and Russian. The description will be given on the assumption that it may be any language, but is not limited to this.
- the origin language of the first character string is a language that determines how to read the first character string.
- the origin language is French.
- the origin of the first character string is French.
- the language of origin of the first character string may be a language determined by a person who uses the first character string, or may be inevitably determined by the contents represented by the first character string.
- the origin language does not mean a language that is the origin of a certain character string, for example, English “etc” originates in Latin “et cetera”.
- the terminal device 200 After displaying the input screen FI1 of FIG. 2A, the terminal device 200 inputs various signals according to the user's operation to the transliteration processing system 1, and generates an input pair based on the input signals. Information representing the input pair is transmitted to the transliteration processing apparatus 100. Thereafter, the terminal device 200 receives the information returned from the transliteration processing device 100, and displays a result display screen FO1 as shown in FIG. 2B based on the received information. Thereafter, the terminal device 200 displays the first character string and the second character string constituting the input pair and the probability that the second character string is a transliteration of the first character string on the result display screen FO1. To do.
- the Japanese DB server 301 and the foreign language DB server 302 have a database that manages information representing an explanation page of an online encyclopedia that an unspecified user is permitted to edit, such as Wikipedia (registered trademark) and DBpedia, for example.
- the Japanese DB server 301 and the foreign language DB server 302 publish information managed by the database on the Internet.
- the Japanese DB server 301 manages and publishes information representing Japanese explanation pages
- the foreign language DB server 302 manages and publishes information representing foreign language comment pages.
- the Japanese commentary page published by the Japanese DB server 301 includes a URL (Uniform Resource Locator) of a foreign language commentary page (hereinafter referred to as a corresponding page) explaining the same thing or person as the Japanese commentary page. Are listed.
- a URL Uniform Resource Locator
- the transliteration processing apparatus 100 is a server machine in which web server software is installed.
- the transliteration processing device 100 receives information representing an input pair from the terminal device 200, calculates a probability that the input pairs represented by the received information are transliterated with each other, and represents the calculated probability To the terminal device 200.
- the transliteration processing apparatus 100 includes a CPU (Central Processing Unit) 100a, a ROM (Read Only Memory) 100b, a RAM (Random Access Memory) 100c, a hard disk 100d, a media controller 100e, a LAN card (Local Area). Network) 100f, video card 100g, LCD (Liquid Crystal Display) 100h, keyboard 100i, speaker 100j, and mouse 100k.
- CPU Central Processing Unit
- ROM Read Only Memory
- RAM Random Access Memory
- the CPU 100a performs overall control of the transliteration processing apparatus 100 by executing software processing according to a program stored in the ROM 100b or the hard disk 100d.
- the RAM 100c temporarily stores data to be processed when the CPU 100a executes the program.
- the hard disk 100d is an information storage unit that stores a table storing various data.
- the transliteration processing apparatus 100 may include a flash memory instead of the hard disk 100d.
- the media controller 100e reads various data and programs from recording media including flash memory, CD (Compact Disc), DVD (Digital Versatile Disc), and Blu-ray Disc (registered trademark).
- the LAN card 100 f transmits and receives data to and from the terminal device 200, the Japanese DB server 301, and the foreign language DB server 302 that are connected via the communication network 10.
- the keyboard 100i and the mouse 100k input signals according to user operations.
- the LAN card 100f constitutes an information acquisition unit, a reception unit, and a probability output unit.
- the video card 100g draws (that is, renders) an image based on the digital signal output from the CPU 100a, and outputs an image signal representing the drawn image.
- the LCD 100h displays an image according to the image signal output from the video card 100g.
- the transliteration processing apparatus 100 may include a PDP (Plasma Display Panel) or an EL (Electroluminescence) display instead of the LCD 100 h.
- the speaker 100j outputs sound based on the signal output from the CPU 100a.
- the CPU 100a of the transliteration processing apparatus 100 functions as a training set generation unit 101 as shown in FIG. 5 by executing the training set generation processing of FIG. 4 using the hardware of FIG. Further, the CPU 100a functions as the information acquisition unit 102 in cooperation with the LAN card 100f. Further, the CPU 100a functions as the information storage unit 103 in cooperation with the hard disk 100d.
- the information storage unit 103 stores a basic transliteration table as shown in FIG. 6A.
- the basic transliteration table of FIG. 6A is a table used to generate a third transliteration table from the first transliteration table shown in FIGS. 6B to 6D.
- the first to third transliteration tables will be described later.
- the basic transliteration table of FIG. 6A the information representative of the original spelling string s n consisting spelling in any source language, object spelling characters consisting spelling was transliterated the spelling string s n to a predetermined target language representing the information representing the column t n, the original spelling string s n and the object spelling string t n and the pair (hereinafter, referred transliteration pairs) number that identifies the (hereinafter referred transliteration pair number) n
- t n ) for translating the original spelling character string s n to the target spelling character string t n are stored in association with each other.
- a transliteration pair identified by a transliteration pair number “1” is composed of an original spelling character string “flextime” and a target spelling character string “flextime”.
- a set including transliteration pairs represented by a plurality of pieces of information stored in the basic transliteration table of FIG. 6A is referred to as a training set.
- the training set generation unit 101 sets the training set as an empty set and represents the number of transliteration pairs added to the training set (this matches the transliteration pair number). By substituting the value “1” for the variable n, the variable n is initialized (step S01).
- the information acquisition unit 102 in FIG. 5 acquires, from the Japanese DB server 301, information representing an arbitrary explanation page constituting the Japanese online encyclopedia, which has not yet been acquired (Step S1). S02).
- the training set generation unit 101 acquires a Japanese character string described in the comment item on the comment page from the comment page represented by the acquired information (step S03).
- the training set generation unit 101 determines whether the acquired Japanese character string is a transliteration of a character string of another language (that is, a foreign language). It is determined based on whether or not it is composed only of (step S04). This is because in Japanese, foreign words are written only in katakana.
- step S04 when it is determined that the acquired Japanese character string is not transliterated (step S04; No), the training set generation unit 101 returns to step S02 and repeats the above processing. On the other hand, when it is determined that the acquired Japanese character string is transliterated (step S04; Yes), the training set generation unit 101 acquires the acquired Japanese character string. Whether or not is a proper noun is determined (step S05).
- the determination method in step S05 is a method for determining whether or not the acquired Japanese character string matches the proper noun represented by the information stored in advance in the information storage unit 103, for example. Alternatively, other known methods may be used.
- step S05 If it is determined in step S05 that the acquired Japanese character string is not a proper noun (step S05; No), the training set generation unit 101 repeats the above-described processing from step S02.
- the comment item on the comment page contains a character string that represents a proper noun such as the name of a thing or the name of a person explained on the comment page, so the character string obtained from the comment item is not a proper noun. This is because there is a high possibility that the character string has not been correctly acquired.
- step S05 If it is determined in step S05 that the acquired Japanese character string is a proper noun (step S05; Yes), the training set generation unit 101 reads the corresponding Japanese character page from step S02. The URL of the corresponding page corresponding to the Japanese explanation page (that is, the corresponding page of the online encyclopedia in a foreign language) is extracted. Next, the training set generation unit 101 acquires information representing the page from the foreign language DB server 302 based on the extracted URL (step S06).
- the Japanese commentary page contains the URL of the corresponding page in a foreign language that explains the same thing or person as the Japanese commentary page.
- the URL of the corresponding page described in advance is extracted, the present invention is not limited to this.
- the training set generation unit 101 is a subdomain included in the URL of the Japanese explanation page, and the subdomain for identifying the country in which the Japanese DB server 301 is installed is changed from “jp” to “en”, “ The URL of the corresponding page in the foreign language may be acquired by changing to “de” or “fr”.
- the training set generation unit 101 acquires a foreign language character string described in the comment item on the comment page from the comment page represented by the acquired information (step S07).
- the training set generation unit 101 removes the parenthesis and the character string enclosed in the parentheses from the acquired Japanese character string and foreign language character string. For example, when there are a plurality of celebrities with the same name, such as Yamada xx (comedy entertainer), noise such as explanatory notes attached in parentheses after the name is removed.
- the training set generation unit 101 After that, the training set generation unit 101, a string of Japanese that have been obtained in step S03 to the original spelling string s n, for the purpose spelling string t n a string of the acquired foreign language in step S07. Next, the training set generation unit 101 sets the pair of the original spelling character string s n and the target spelling character string t n as a transliteration pair (step S08), and then adds the transliteration pair to the training set. (Step S09). Thereafter, the training set generation unit 101, to basic transliteration table of FIG. 6A, the information representing the information representing the transliteration number n of the transliteration pair, the original spelling string s n constituting the transliteration pair, Save associates the information representing the desired spelling string t n, a (step S10).
- the training set generation unit 101 increments the value of the variable n representing the transliteration number by a value “1” (step S11), and then determines whether or not the value of the variable n is equal to or greater than a predetermined value m (step S11).
- Step S12 Information representing the predetermined value m is stored in the information storage unit 103 in FIG. In this embodiment, the description will be made assuming that the predetermined value m is “11,000”. However, the present invention is not limited to this, and a person skilled in the art can determine a suitable value by experiment.
- step S12 If it is determined in step S12 that the value of the variable n is less than the predetermined value m (step S12; No), the training set generation unit 101 repeats the above-described processing from step S02. On the other hand, when it is determined that the value of the variable n is equal to or greater than the predetermined value m (step S12; Yes), the training set generation unit 101 ends the execution of the training set generation process.
- the CPU 100a of the transliteration processing apparatus 100 executes the transliteration process of FIG. 7 by using the hardware of FIG. 3, thereby generating the first generation unit 105a and the second generation unit 105b as shown in FIG. It further functions as an updating unit 107 having a unit 105, an initialization unit 106, an expected value calculation unit 107a, a determination unit 107b, and a maximization unit 107c.
- the CPU 100a further functions as the probability output unit 108 in cooperation with the LAN card 100f.
- the information storage unit 103 stores a basic rewrite table as shown in FIG. 8A.
- the basic rewrite table in FIG. 8A is a table used for creating the third rewrite table from the first rewrite table in FIGS. 8B to 8D.
- the first to third rewrite tables will be described later.
- the basic rewrite table of FIG. 8A original spelling string s n fragment represented by information stored in the basic transliteration table of FIG. 6A (hereinafter, referred to as the original fragments) and information representative of the alpha, the original spelling characters Information representing a fragment (hereinafter referred to as transliteration fragment) ⁇ corresponding to the original fragment ⁇ , a fragment of the target spelling character string t n obtained by transliterating the sequence, the original fragment ⁇ , the transliteration fragment ⁇ , Information indicating a fragment pair number j for identifying a pair (hereinafter referred to as a fragment pair) and information indicating a probability that the original fragment ⁇ is rewritten to the transliterated fragment ⁇ for transliteration (hereinafter referred to as a rewriting probability).
- the original fragment and the transliteration fragment may both be one character or a character string consisting of two or more characters.
- the fragment pair identified by the fragment pair number “1” is composed of an original fragment ⁇ “ ⁇ f” and a transliterated fragment ⁇ “ ⁇ fu”.
- the fragment pair identified by the fragment pair number “5” is composed of the original fragment ⁇ “get $” and the transliterated fragment ⁇ “Je $”, and is identified by the fragment pair number “6”.
- the fragment pair is composed of an original fragment ⁇ called “get $” and a transliterated fragment ⁇ called “get $”. In this way, different transliteration fragments “Je $” and “get $” are associated with the same original fragment “get $” because the two transliteration pairs described below are associated with the training set. Because it was included.
- One is a transliteration pair of an original spelling character string s n ending with “get $” and a target spelling character string t n transliterated based on a phoneme obtained by reading the original spelling character string s n in French. It is.
- the other is transliteration of the original spelling character string s n ending with “get $” and the target spelling character string t n transliterated based on the phoneme obtained by reading the original spelling character string s n in English. It is a pair. Note that “ ⁇ ” is a symbol indicating that the character immediately following is the beginning of the character string, and “$” is a symbol indicating that the immediately preceding character is the end of the character string.
- the information storage unit 103 stores a rewrite count table as shown in FIG.
- the rewrite count table of FIG. 9 information representing the transliteration pair number n, information representing the fragment pair number j, and information representing the rewrite count f n ( ⁇ ⁇ ⁇ ) are stored in association with each other.
- the number of rewriting f n ( ⁇ ⁇ ⁇ ) is the original spelling string s n constituting transliteration pair identified with the number n, for transliterated the purpose spelled string tn constituting the transliteration pairs Is the number of times the original fragment ⁇ of the fragment pair identified by the fragment pair number j is rewritten to the transliterated fragment ⁇ .
- the fragment pair number “2” The number of times of rewriting that the original fragment “le” constituting the fragment pair identified by “” is rewritten into the transliterated fragment “le” is “1”.
- the original fragment “le” of the fragment pair identified by the fragment pair number “3” is transliterated. The number of times of rewriting to “le” is “0”.
- the number of times of rewriting the original fragment “ma” of the pair with the transliteration fragment “ma” is two.
- the accepting unit 104 in FIG. 5 accepts an input pair represented by information transmitted from the terminal device 200 (step S31).
- the generation unit 105 regards the received input pair as a transliteration pair. Thereafter, the generation unit 105 adds the transliteration pair to the training set, sets the transliteration pair number of the transliteration pair to m + 1, and sets a variable N representing the number of transliteration pairs belonging to the training set to m + 1.
- the generation unit 105 adds and stores information representing the transliteration pair number N and information representing the transliteration pair in the basic transliteration table of FIG. 6A (step S32).
- the first generation unit 105a included in the generation unit 105 executes a first generation process as illustrated in FIG. 10 (step S33).
- the first generation unit 105a uses the acquired transliteration pair after acquiring the transliteration pair included in the training set by referring to the basic transliteration table of FIG. 6A.
- the rewrite count table of FIG. 9 is generated (step S51).
- the first generation unit 105a extracts all transliteration pairs from the training set.
- the first generation unit 105a for all transliteration pairs taken out, and add the special character " ⁇ " at the beginning of the original spelling string s n constituting the transliteration pairs, special trailing characters Add “$”.
- the first generation unit 105a generates an original fragment ⁇ by dividing the string s n which special characters are added.
- First generating unit 105a by dividing original spelling string s n division method S of all street, to generate the ⁇ original fragment of the entire street.
- the first generation unit 105a sets a set including all the division methods S of the original spelling character string s n as Part (s n ).
- the first generation unit 105a generates a set Part (t n ) including all the specified division methods T as elements after specifying all the division methods T of the target spelling character string t n. .
- the first generation unit 105a acquires the entire transliteration fragment ⁇ by dividing the target spelling character string t n by the entire division method T.
- the first generation unit 105a generates all combinations of the original fragment ⁇ and the transliteration fragment ⁇ , and calculates the rewrite frequency f n ( ⁇ ⁇ ⁇ ) for each generated combination.
- the first generation unit 105a converts the information indicating the calculated rewrite frequency f n ( ⁇ ⁇ ⁇ ) into the rewrite frequency table of FIG. 9, information indicating the transliteration pair number n, and the original fragment ⁇ . And the information indicating the fragment pair number j of the fragment pair composed of the character fragment ⁇ .
- the first generation unit 105a After step S51 in FIG. 10, the first generation unit 105a generates the basic rewrite table in FIG. 8A based on the rewrite count table in FIG. 9 (step S52). Specifically, the first generation unit 105a uses the calculated rewrite frequency f n ( ⁇ ⁇ ⁇ ) in the following equation (1) to rewrite the original fragment ⁇ into the transliteration fragment ⁇ . P ( ⁇ ⁇ ⁇ ) is calculated.
- the first generation unit 105a corresponds the information indicating the calculated rewrite probability P ( ⁇ ⁇ ⁇ ) to the information indicating the fragment pair number j of the fragment pair composed of the original fragment ⁇ and the transliteration fragment ⁇ . In addition, it is stored in the basic rewriting table of FIG. 8A.
- the first generation unit 105a regenerates the basic transliteration table of FIG. 6A based on the basic rewrite table of FIG. 8A (step S53), and then ends the execution of the first generation process. .
- the first generation unit 105a pays attention to an unprocessed transliteration pair among transliteration pairs belonging to the training set.
- the first generation unit 105a includes the original spelling character string s n and the target spelling character string t n constituting the noticed transliteration pair (hereinafter referred to as the noticed transliteration pair), and the calculated rewrite probability P ( ⁇ ⁇ ⁇ ) is used in the following equation (2) to execute the ⁇ method on the target transliteration pair.
- the noticed transliteration pair the original spelling character string s n and the target spelling character string t n constituting the noticed transliteration pair
- the calculated rewrite probability P ( ⁇ ⁇ ⁇ ) is used in the following equation (2) to execute the ⁇ method on the target transliteration pair.
- T represents a division method of the target spelling character string t n
- Part (t n ) is a set having the whole division method T as an element.
- S represents a dividing method of the original spelling character string s n
- Part (s n ) is a set having the entire dividing method S as an element.
- represents the number of the original fragments ⁇ produced when dividing the original spelling string s n division method S.
- ⁇ i represents the i-th original fragment generated when the original spelling character string s n is divided by the dividing method S
- ⁇ i represents the target spelling character string t n by the dividing method T. This represents the i-th transliteration fragment from the top generated when divided.
- the first generation unit 105a By executing the ⁇ method, the first generation unit 105a causes the transliteration probability P ⁇ (s n
- the accepting unit 104 in FIG. 5 accepts the number K of languages represented by the information transmitted from the terminal device 200 (step S34).
- language number K is the original spelling string s n constituting each multiple transliteration pairs belonging to the training set, potentially the number of classes depending on origin language (i.e., explicit instead) classifies Is a value representing Specifically, taking as an example a case where a plurality of original spelling strings s n included in the training set have one of English, French, Spanish, German, Polish, and Russian as the origin language. explain.
- the number of accepted languages K is a value “7”
- the plurality of original spelling strings s n are respectively in English class, French class, German class, Polish according to the origin language. It is often classified into seven classes: a word class and a Russian class.
- the description is made assuming that the number K of latent languages is “3”, but the number is not limited to this.
- the “3” latent languages are identified by language numbers “1”, “2”, and “3”, respectively.
- the latent languages identified by the language numbers “1”, “2”, and “3” are referred to as a first language, a second language, and a third language, respectively.
- the language number z is a latent variable, and the specific number of the latent language identified by the language number z is not specified.
- the second generation unit 105b in FIG. 5 executes a second generation process as shown in FIG.
- the second generation unit 105b reads the basic rewrite table of FIG. 8A (step S61).
- the second generation unit 105b generates K rewrite tables from the first rewrite table to the Kth rewrite table by copying K read basic rewrite tables (step S62).
- the second generation unit 105b includes information representing the number j for identifying the fragment pair stored in the basic rewrite table in FIG. 8A, information representing the original fragment ⁇ constituting the fragment pair, and the fragment.
- a first rewriting table is generated in which information representing the transliteration fragment ⁇ constituting the pair and information representing the rewriting probability P ( ⁇ ⁇ ⁇
- z 1) are stored in association with each other.
- the second generation unit 105b stores a second rewrite table in which information representing the rewrite probability P ( ⁇ ⁇ ⁇
- z 2) is stored, and information representing the rewrite probability P ( ⁇ ⁇ ⁇
- z 3).
- a third rewrite table in which is stored.
- step S62 the second generation unit 105b reads the basic transliteration table of FIG. 6A (step S63). After that, the second generation unit 105b generates K transliteration tables from the first transliteration table to the Kth transliteration table by copying K read basic transliteration tables (step S64). Then, the execution of the second generation process is terminated.
- the second generation unit 105b receives the information representative of the number n identifying transliteration pairs stored in basic transliteration table, and information representing an original spelling string s n constituting the transliteration pairs 6B, the information indicating the target spelling character string t n constituting the transliteration pair and the information indicating the transliteration probability P (t n
- s n , z 1) are stored in association with each other.
- a first transliteration table as shown is generated.
- the second generation unit 105b stores a second transliteration table in which information representing transliteration probability P (t n
- s n , z 2) as illustrated in FIG. 6C is stored, and as illustrated in FIG. 6D. And a third transliteration table in which information representing the transliteration probability P (t n
- s n , z 3) is stored.
- s n, z 1), transliteration probability P (t n
- s n, z 2), and transliteration probability P (t n
- the second generation unit 105b converts the transliteration probability P (t n
- s n , z 1) associated with information representing the same transliteration pair number n, and the transliteration probability P (t n
- s n , z 2) and the transliteration probability P (t n
- s n , z 3) with respect to the transliteration probability P ⁇ (t n
- Each random value is set to a value obtained by addition, subtraction, multiplication, division, or the like.
- the present invention is not limited to this, and the transliteration probability P (t n
- s n , z 1).
- s n ) may be set to a different value by a predetermined amount for each language.
- the initialization unit 106 in FIG. 5 executes an initialization process for initializing parameters used in the execution of the EM algorithm described later (step S36). Specifically, the initialization unit 106, among the N number of original scripts string s n constituting transliteration pairs belonging to the training set, the original spelling of the potential language identified by the language number k originate Language A weight parameter ⁇ k representing the ratio of the character string s n is initialized to a predetermined value “1 / K”.
- Information representing a predetermined value is stored in the information storage unit 103.
- the initial value of the weight parameter ⁇ k represented by the information is not limited to “1 / K”, and those skilled in the art can determine a suitable value by experiment.
- the weight parameter ⁇ 1 is referred to as the weight parameter of the first transliteration table shown in FIG. 6B
- the weight parameter ⁇ 2 is referred to as the weight parameter of the second transliteration table shown in FIG. 6C
- the weight parameter ⁇ 3 is referred to as a weight parameter of the third transliteration table shown in FIG. 6D.
- the expected value calculation unit 107a in FIG. 5 executes the E step of the EM algorithm (step S37).
- the E step is composed of a first E step to a third E step.
- the expected value calculation unit 107a executes only the second E step and the third E step from the first E step to the third E step constituting the E step.
- the expected value calculation unit 107a executes the first E step to the third E step in order.
- the 1E step performs the ⁇ method using the rewriting probabilities P ( ⁇ ⁇ ⁇
- z 1) to P ( ⁇ ⁇ ⁇
- z 3), thereby obtaining the transliteration probability P (t n
- s n , z 3) from
- s n , z 1).
- the process corresponding to the first 1E step is the process of step S53 in FIG. 10 and step S64 in FIG. 11 and has already been executed.
- the expected value calculation unit 107a pays attention to the first language with the language number “1” and sets the first language as the attention language.
- the expected value calculation unit 107a refers to the rewrite probability P ( ⁇ ⁇ ⁇
- z 1) from the first rewrite table of FIG. 8B corresponding to the target language. Thereafter, the expected value calculation unit 107a pays attention to one of the unprocessed transliteration pairs represented by the information stored in the first transliteration table of FIG. 6B.
- the expectation value calculation unit 107a includes the original spelling character string s n and the target spelling character string t n constituting the noticed transliteration pair (hereinafter referred to as the noticed transliteration pair), and the rewriting probability P ( ⁇ ⁇ ⁇
- Z 1) is used in the following equation (3) to execute the ⁇ method.
- the expected value calculation unit 107a repeats the above process until there is no unprocessed transliteration pair, and then repeats the above process using the second language identified by the language number “2” as the target language.
- the expected value calculation unit 107a repeats the above process using the third language identified by the language number “3” as the target language, and then ends the first E step.
- the expected value calculation unit 107a calculates the expected value ⁇ nk (that is, E [z nk ]) of the random variable z nk with respect to all transliteration pair numbers n and all language numbers k by the following formula ( 4).
- This random variable z nk has a value “1” when the original spelling character string s n of the transliteration pair identified by the transliteration pair number n originates from the latent language identified by the language number k, and is not so. In this case, it is a variable that has the value “0”.
- the expected value ⁇ nk is a weighted transliteration probability obtained by weighting the transliteration probability P (t n
- s n , z 1) with the weight parameter ⁇ k , and the origin language of the original spelling character string s n is It is also the probability of being a latent language identified by language number k (hereinafter referred to as origin probability).
- the expected value calculation unit 107a calculates the expected value Q of the likelihood function L using the expected value ⁇ nk .
- the likelihood function L is a function for calculating likelihood, and is represented by the following equation (5).
- the likelihood calculated by the likelihood function L is obtained by assigning a plurality of transliteration pairs as elements of the training set to the language numbers “1” to “3”.
- 6 represents the likelihood of the first to third transliteration tables shown in FIGS. 6B to 6D in the transliteration model potentially classified based on the three latent languages.
- step S38 the determination unit 107b in FIG. 5 determines whether or not the expected value Q of the likelihood function L has been calculated for the first time (that is, whether the E step has been executed for the first time) (step S38). .
- the update unit 107 determines that the expected value Q of the likelihood function L has not been calculated for the first time this time (step S38; No)
- the expected value of the likelihood function L previously calculated in step S37.
- the absolute value of the amount of change of the likelihood function L calculated this time to the expected value Q is calculated.
- the updating unit 107 determines whether or not the calculated absolute value of the change amount is equal to or greater than a predetermined value Z (step S39).
- information representing the predetermined value Z is stored in the information storage unit 103 of FIG. 5, and a suitable value of the predetermined value Z can be determined by experiments by those skilled in the art.
- the determination is performed based on the absolute value of the change amount, but the determination may be performed based on the ratio instead of the absolute value. For example, it is determined whether the ratio of the expected value Q of the likelihood function L calculated this time to the expected value Q of the likelihood function L calculated last time is greater than or equal to a predetermined value Z, percentage of the expected value gamma nk calculated for the current against the expected value gamma nk was that may be subjected to a determination whether or not a predetermined value or more Z.
- step S38 when the updating unit 107 determines that the expected value Q of the likelihood function L is calculated for the first time this time (step S38; Yes), or the absolute value of the change amount of the expected value Q is equal to or greater than the predetermined value Z. (Step S39; Yes), the maximization unit 107c in FIG. 5 executes the M step that maximizes the expected value Q of the likelihood function L (step S40).
- the maximization unit 107c calculates the number of rewrites f n ( ⁇ ⁇ ⁇ ) represented by the information stored in the rewrite number table in FIG. 9 and step S37 in FIG. 7 (that is, E step).
- the expected value ⁇ nk is used in the following formulas (6) and (7).
- the maximization unit 107c sets the weight parameter ⁇ k * that maximizes the expected value Q of the likelihood function L and the transliteration probability P (t n
- s n , z k) * to all The transliteration pair number n and the language number k are respectively calculated.
- step S41 the update unit 107 executes parameter update processing (step S41).
- the updating unit 107 uses the weight parameter ⁇ k used for calculating the expected value Q of the likelihood function L in step S37, and the optimum weight parameter ⁇ k * (that is, calculated in step S40). And an optimal solution that maximizes the expected value Q of the likelihood function L). Further, the updating unit 107 calculates the transliteration probability P (t n
- s n , z 1) from the transliteration probabilities P (t n
- s n , z 1) stored in the first transliteration table to the third transliteration table of FIGS.
- s n , z 1) * to P (t n
- s n , z 3) * , respectively. Update to information. Thereafter, the above process is repeated from step S37.
- transliteration probability output processing is executed (Step S42).
- the probability output unit 108 in FIG. 5 converts from transliteration probabilities P (t N
- s N , z 1) to P (t Information representing t N
- s N , z 3) is referred to from the first transliteration table of FIG. 6B to the third transliteration table of FIG. 6D.
- the probability output unit 108 displays information representing the maximum transliteration probability from the referenced transliteration probability P (t N
- s N , z 1) to P (t N
- s N , z 3). Output (ie, transmit) to the terminal device 200. After executing the transliteration probability output process, the probability output unit 108 ends the execution of the transliteration process.
- step S62 of FIG. 11 the original fragment “get $” constituting the fragment pair identified by the fragment pair number “5” stored in the first rewriting table of FIG. 8B is rewritten to the transliteration fragment “Je $”.
- z 2) and P ( ⁇ ⁇ ⁇
- z 3).
- s 2 , z 2) and P (t 2
- higher than s 2 , z 3).
- weighting transliteration probability gamma 21 calculated by the above equation (4) is, as shown in FIG. 12B, it is higher than the other weighting transliteration probability gamma 22 and gamma 23.
- z 2) and P ( ⁇ ⁇ ⁇
- z 3) are updated to higher values. Therefore, the transliteration probability P (t 2
- s 2 , z 1) is greater than the other transliteration probabilities P (t 2
- s 2 , z 2) and P (t 2
- s 2 , z 3). Will be even higher.
- the first transliteration table corresponding to the language has been described.
- the number of latent languages is not specified, for example, an original spelling character string s 2 “whose origin language is a latent language such as French is used.
- s n , z k). Thereafter, the calculation of the transliteration probability P (t n
- the origin probability (that is, the expected value of the random variable z nk , using the transliteration probability P (t n
- s n , z k) and the weight parameter ⁇ k , and the origin probability ⁇ nk is calculated. Thereafter, the transliteration probability P is maximized so that the expected value of the likelihood function L calculated using not only the transliteration probability P (t n
- s n , z k) but also the origin probability ⁇ nk is maximized. Not only (t n
- s n , z k) but also the weight parameter ⁇ k is updated. For this reason, the expected value of the likelihood function L is further maximized compared to the conventional case. Therefore, even if it is unclear how many words the original spelling character string s n that is the target of transliteration originates, it is determined whether the translated target spelling character string t n is a transliteration of the target. It can be identified with high accuracy.
- t n ) included in one basic transliteration table is generated by executing the ⁇ method from one basic rewriting table.
- K transliteration tables corresponding to the K languages are generated by changing the generated transliteration probability P ⁇ (s n
- the expected value Q of the likelihood function L can be reliably maximized as compared with the prior art.
- the transliteration processing apparatus 100 when the first character string and the second character string are received, a probability that the received second character string is a transliteration of the first character string is output. Therefore, even if it is unclear how many words the first character string originates, if the first character string and the second character string are accepted, the second character string is a transliteration of the first character string. A certain probability can be calculated and output more accurately than in the past. Therefore, even if the user who uses the transliteration processing apparatus 100 does not know the origin language of the first character string, the first character string and the second character string that is considered to be a transliteration of the first character string, Is input to the transliteration processing apparatus 100, the transliteration probability between the first character string and the second character string can be known. For this reason, the user can easily and accurately determine whether or not the second character string is a correct transliteration of the first character string based on the transliteration probability.
- the first character string not included in the training set is accepted, and the transliteration probability between the accepted first character string and the second character string is calculated. For this reason, even if it is a pair (namely, input pair) of the 1st character string and the 2nd character string which are not included in the training set beforehand, the 2nd character string transliterates the 1st character string. Probability can be calculated and output more accurately than before.
- the number of languages K represented by the information transmitted from the terminal device 200 in step S34 in FIG. 7 is accepted.
- the transliteration processing device 100 is preset with a value of K. Also good.
- the number of languages K to be set is not limited to one and may be plural. For example, if three values “3”, “4”, and “5” are set as K, the second character string is the second character string based on the highest transliteration probability among the transliteration probabilities obtained by each. The user can determine whether or not the character string is a correct transliteration.
- the generation unit 105 of FIG. 5 generates the basic transliteration table of FIG. 6A in which information representing the transliteration probability P ⁇ (s n
- the update unit 107 in FIG. 5 has been described as not performing the first E step when the step S37 in FIG. 7 (that is, the E step) is executed for the first time.
- the first E step is a step of calculating the transliteration probability P (t n
- s n , z k) by executing the ⁇ method using the third rewrite table from the first rewrite table.
- the generation unit 105 in FIG. 5 stores information representing the transliteration probability P ⁇ (s n
- the generation unit 105 generates a third basic transliteration table from the first basic rewriting table based on the basic rewriting table in step S62 of FIG.
- the updating unit 107 in FIG. 5 executes the steps from the 1E step to the 3E step in the first execution of step S37 in FIG.
- the rewriting probability P ( ⁇ ⁇ ⁇ ) associated with the information representing j may be set to a value obtained by adding, subtracting, multiplying, or dividing a random value, or for each of K languages. Alternatively, different values may be set for each predetermined amount.
- the terminal device 200 in FIG. 1 displays the input screen FI2 as shown in FIG. 13A instead of the input screen FI1 in FIG. 2A, so that the first character string composed of spellings in an arbitrary origin language is displayed. Displays a prompt to input.
- the terminal device 200 inputs various signals according to user operations to the transliteration processing system 1, generates a first character string based on the input signal, and represents the generated first character string. Information is transmitted to the transliteration processing apparatus 100. Thereafter, the terminal device 200 receives the information returned from the transliteration processing device 100.
- the terminal device 200 displays a result display screen FO2 as shown in FIG.
- the terminal device 200 based on the received information, the first character string, the candidate character string in the predetermined target language to which the first character string can be transliterated, and the first character string as the candidate
- the probability of transliteration into a character string is displayed in a list on the result display screen FO2 in descending order of the probability.
- the terminal device 200 may display a list of candidate character strings in descending order of the probability.
- the terminal device 200 may display only the first character string, the candidate character string, and the probability that have the highest probability.
- the terminal device 200 may display only the highest probability, one character string, a candidate character string, and the probability only when the highest probability exceeds a predetermined threshold.
- the CPU 100a of the transliteration processing apparatus 100 executes the transliteration process of FIG. 14 instead of FIG. 7 using the hardware of FIG. Units 101 to 107 and function as candidate output unit 109.
- the accepting unit 104 in FIG. 15 accepts the first character string represented by the information transmitted from the terminal device 200 (step S71).
- generation part 105 performs the 1st production
- the generation unit 105 generates a plurality of candidate character strings of the accepted first character string (step S73).
- the generation unit 105 divides the first character string by an arbitrary dividing method S, and acquires all fragment character strings or fragment characters (hereinafter referred to as fragments). Next, the generation unit 105 selects all combinations (that is, all arrangements) of the same fragments as the acquired fragments among the original fragments ⁇ constituting the fragment pairs represented by the information stored in the basic rewriting table of FIG. 8A. Is generated. Next, the production
- the generation unit 105 combines the transliteration fragments ⁇ constituting the fragment pairs represented by the acquired information in accordance with the combination order (that is, the arrangement order) of the original fragments ⁇ that form a pair of the transliteration fragments ⁇ . In this way, the generation unit 105 generates all candidate character strings that can be considered as transliterations of the first character string.
- the generation unit 105 After step S73 in FIG. 14, the generation unit 105 generates, for each of the plurality of generated candidate character strings, the first character string received in step S71 and the candidate character string considered as a transliteration of the first character string.
- a pair (hereinafter referred to as a candidate pair) is generated (step S74).
- the generation unit 105 regards the candidate pair as a transliteration pair and adds the candidate pair to the training set, and increases the value N of the number of transliteration pairs included in the training set by the number of added candidate pairs.
- the generation unit 105 associates the information representing the candidate pair with the information representing the number n for identifying the candidate pair, and additionally stores the information in the basic transliteration table in FIG. 6A (step S75).
- step S76 to step S83 in FIG. 14 which is the same as the processing from step S34 to step S41 in FIG. 7, is executed.
- step S81 the determination unit 107b determines that the absolute value of the amount of change from the previously calculated expected value Q of the likelihood function L to the expected value Q of the likelihood function L calculated this time is calculated.
- step S84 the candidate output unit 109 in FIG. 15 converts from the transliteration probabilities P (t N
- s N , z 1) associated with information representing the number n identifying each candidate pair to P (t N
- S N , z 3) is referred to from the first transliteration table of FIG. 6B to the third transliteration table of FIG.
- the candidate output unit 109 includes information representing each candidate pair and the transliteration probability P (t N
- s N , z 1) to P (t N
- s N , z 3). To generate candidate list information associated with information representing the maximum transliteration probability. Thereafter, the candidate output unit 109 sorts a list of candidate character strings represented by the generated candidate list information (hereinafter referred to as a candidate list) in ascending order or descending order based on the maximum transliteration probability. Thereafter, the candidate output unit 109 outputs (that is, transmits) the sorted list to the terminal device 200 in order from the top of the list, and then ends the process of step S84.
- a candidate list a list of candidate character strings represented by the generated candidate list information
- a plurality of candidate character strings that are considered to be transliterations of the first character string are converted into transliteration probabilities that the candidate character strings are transliterations of the first character string.
- Output based on For this reason, even if it is unclear how many words the first character string originates, if the first character string is accepted, a plurality of candidate character strings can be output in the order of high or low transliteration probability. Therefore, even if the user who uses the transliteration processing apparatus 100 does not know the origin language of the first character string, as long as the first character string is input to the transliteration processing apparatus 100, the first character string A plurality of candidate character strings that can be considered as transliteration of.
- the first character string not included in the training set is received, the candidate character string of the received first character string is generated, and the transliteration probability with the generated candidate character string is calculated. To do. For this reason, even if it is the 1st character string which is not included in the training set beforehand, if a 1st character string is received, a some candidate character string can be output in order with a high or low transliteration probability.
- the transliteration processing system 1 of Modification 3 does not have the foreign language DB server 302 of FIG. 1, but has a search engine 400 as shown in FIG.
- the search engine 400 includes information representing a search keyword, information representing a URL of a Web (World Wide Web) page in which content including the search keyword is described, and the keyword included in the content described in the Web page. Is stored in association with information representing a snippet that is a portion including.
- the search engine 400 retrieves a plurality of pieces of information representing snippets stored in association with the received information representing the search keyword. Thereafter, the search engine 400 returns information representing the search result screen FS including the snippets SP1 and SP2 as shown in FIG.
- the transliteration processing apparatus 100 of Modification 3 determines that the character string acquired from the comment item of the Japanese online encyclopedia is a proper noun (step S05; Yes) ), Step S06 is not executed, but step S91 of FIG. 18 is executed.
- step S91 the transliteration processing apparatus 100 outputs the keyword and the search request to the search engine 400 of FIG. 16 using the Japanese character string as a search keyword.
- the information acquisition unit 102 of the transliteration processing apparatus 100 shown in FIG. 5 acquires a plurality of pieces of information representing snippets searched based on the search keyword from the search engine 400 (step S92).
- the training set generation unit 101 of the transliteration processing apparatus 100 illustrated in FIG. 5 is a predetermined number of times before or after the search keyword described in the snippet (that is, the character string described in the Japanese commentary item). Is acquired (step S93). Thereafter, the training set generation unit 101 determines whether or not the word is represented by a character string composed of spellings in a foreign language based on whether or not the acquired word is represented by a character string composed of alphabets. Is discriminated (step S94). At this time, if it is determined that the acquired word is not represented by a character string composed of a spelling in a foreign language (step S94; No), the processing is repeated from step S02 of FIG.
- step S94 if it is determined that the acquired word is represented by a character string composed of a spelling in a foreign language (step S94; Yes), the training set generation unit 101 reads the acquired foreign language character. Using the column as a search keyword, the same processing as in steps S91 and S92 is executed (step S95 and step S96).
- the training set generation unit 101 determines whether or not the acquired word is represented by a character string composed of spellings in Japanese (step S98). Specifically, when the training set generation unit 101 determines that the acquired word is represented by a character string composed of at least one of kanji, hiragana, and katakana, the word is in Japanese It is determined that the character string is composed of spellings. On the other hand, if the training set generation unit 101 determines that the acquired word is represented by a character string that is not composed of any kanji, hiragana, or katakana, the training set generation unit 101 determines that the word is spelled in Japanese. It is determined that the character string is not represented.
- step S98 If it is determined in step S98 that the acquired word is not represented by a character string composed of spellings in Japanese (step S98; No), the processing is repeated from step S02 in FIG.
- step S98; Yes when it is determined that the acquired word is represented by a character string composed of spellings in Japanese (step S98; Yes), the training set generation unit 101 displays Japanese in step S03 of FIG. It is determined whether or not the character string acquired from the comment item of FIG. 18 matches the Japanese character string acquired from the snippet in step 97 of FIG. 18 (step S99). At this time, if it is determined that the two character strings do not match (step S99; No), the processing is repeated from step S02 of FIG.
- step S99 If it is determined in step S99 that the two character strings match (step S99; Yes), the training set generation unit 101 determines the Japanese character string acquired from the Japanese commentary item and the Japanese character string.
- the foreign language character string acquired from the snippet retrieved based on the word character string is used as a transliteration pair (step S100).
- the training set generation unit 101 sets the foreign language character string as the original spelling character string s n and the Japanese character string as the target spelling character string t n (ie, transliteration of the original spelling character string s n ). ).
- the execution of the training set generation process of FIG. 4 is continued from the process of step S09 of FIG. 4 in which the transliteration pair is added to the training set.
- the transliteration processing apparatus 100 acquires a foreign language character string from a snippet searched based on a Japanese character string acquired from a comment item of a Japanese online encyclopedia. Next, the transliteration processing apparatus 100 acquires a Japanese character string from the snippet searched based on the acquired foreign language character string. After that, when these two Japanese character strings match, the transliteration processing apparatus 100 sets the foreign character string as the original spelling character string s n and the Japanese character string as the target spelling character string t n .
- the present invention is not limited to this, and the following modifications can be adopted.
- the transliteration processing apparatus 100 acquires a Japanese character string from a snippet searched based on a foreign language character string acquired from a comment item of a foreign language online encyclopedia. Next, the transliteration processing apparatus 100 acquires a foreign language character string from the snippet searched based on the acquired Japanese character string. Then, transliteration processing device 100, these if the two strings match, and a string of foreign language with the original spelling string s n, a string of Japanese may be for the purpose of spelling string t n a.
- the original spelling character string s n and the target spelling character string t n that are transliterated with each other are often described so as to be positioned in front of and behind each other on the Web page. For this reason, according to these configurations, even when the origin language of the original spelling character string s n is unknown, the original spelling character string s n and the target spelling character string t n can be more accurately and easily compared to the conventional case. The transliteration pair of can be acquired.
- FIG. 6A the purpose spelled string t n, have been illustrated and described as represented in katakana, not limited thereto, e.g., romaji Or other characters including phonetic symbols.
- the transliteration fragment ⁇ has been illustrated and described as being expressed in katakana, but is not limited to this, for example, in Roman letters or phonetic symbols. May be represented.
- the present embodiment and the first to third modifications can be combined with each other.
- the transliteration processing system is configured by a plurality of apparatuses, and realizes the functions according to the present embodiment. It is also possible to provide a transliteration processing system provided with the configuration for the entire system.
- the existing transliteration processing apparatus 100 can be converted into the transliteration processing according to the present embodiment by applying a program. It can also function as the device 100. That is, the transliteration processing program for realizing each functional configuration by the transliteration processing apparatus 100 illustrated in the above embodiment is applied so that a computer (such as a CPU) that controls the existing transliteration processing apparatus can be executed. Thus, it can function as the transliteration processing apparatus 100 according to the present embodiment.
- Such a program distribution method is arbitrary.
- the program can be distributed by being stored in a recording medium such as a memory card, a CD-ROM, or a DVD-ROM, or can be distributed via a communication medium such as the Internet.
- the transliteration processing method according to the present invention can be implemented using the transliteration processing apparatus 100 according to the present invention.
- the transliteration processing method according to the present invention can be implemented using the transliteration processing apparatus 100.
- the present invention is suitable for a server device that provides, in response to a request from a terminal device, a service that specifies whether the character string received from the terminal device is a transliterated character string to be transliterated.
- Communication network 100 Transliteration processing device 100a: CPU 100b: ROM 100c: RAM 100d: hard disk 100e: media controller 100f: LAN card 100g: video card 100h: LCD 100i: keyboard 100j: speaker 100k: mouse 101: training set generation unit 102: information acquisition unit 103: information storage unit 104: reception unit 105: generation unit 105a: first generation unit 105b: second generation unit 106: initialization unit 107: update unit 107a: expected value calculation unit 107b: determination unit 107c: maximization unit 108: probability output unit 109: candidate output unit 200: terminal device 301: Japanese DB server 302: foreign language DB server 400: search engine
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Machine Translation (AREA)
- Document Processing Apparatus (AREA)
- Stored Programmes (AREA)
Abstract
Description
任意の起源言語における綴りからなる原綴り文字列と、当該原綴り文字列を所定の目的言語に翻字した綴りからなる目的綴り文字列と、の翻字対を複数含む訓練集合から、前記原綴り文字列を構成する原断片と、前記目的綴り文字列を構成する翻字断片と、翻字のために当該原断片が当該翻字断片に書き換えられる書換確率との組を複数含む、互いに異なるK個の起源言語に対応したK個の書換表と、前記訓練集合に含まれる翻字対を複数含む、前記K個の起源言語に対応したK個の翻字表とを生成する生成部、
前記訓練集合に複数含まれる翻字対のそれぞれに対して、前記K個の書換表に含まれる書換確率のそれぞれを用いることにより、当該翻字対の原綴り文字列が当該書換表に対応した起源言語を起源とする場合に当該翻字対の目的綴り文字列に翻字される翻字確率を算出してから、当該翻字確率を当該翻字対と対応付けて当該起源言語に対応した翻字表に保存した後に、前記訓練集合が得られた場合における前記K個の翻字表の尤もらしさを表す尤度を算出する尤度関数の期待値であって、当該翻字確率を用いて算出される値を最大化するように、前記K個の書換表のそれぞれに含まれる書換確率を更新した後に、前記翻字確率の算出と前記書換確率の更新とを繰り返す更新部、
を備えることを特徴とする。
前記K個の翻字表それぞれに対応した起源言語を起源とする原綴り文字列が、前記訓練集合に含まれる複数の原綴り文字列に占める割合をそれぞれ示す、前記K個の翻字表の重みパラメタを初期化する初期化部、を更に備え、
前記更新部は、前記訓練集合に複数含まれる翻字対のそれぞれに対して、前記K個の翻字表に含まれる当該翻字確率の前記重みパラメタによる重み付き平均に基づいて、前記原綴り文字列の起源言語が当該翻字表に対応した起源言語である起源確率をそれぞれ算出した後に、当該起源確率をさらに用いて算出される前記尤度関数の期待値を最大化するように、前記K個の重みパラメタを更新した後に、前記起源確率の算出と前記重みパラメタの更新とを繰り返す、としても良い。
前記生成部は、1個の前記書換表を生成した後に、前記生成された1個の書換表に含まれる書換確率を、前記K個の言語毎に変化させることにより、前記K個の言語に対応したK個の書換表を生成する、としても良い。
前記生成部は、1個の前記書換表を用いたαβ法により1個の前記翻字表を生成した後に、前記生成された1個の翻字表に含まれる翻字確率を変化させることにより、前記K個の言語に対応したK個の翻字表を生成し、
前記更新部は、前記K個の翻字表のそれぞれに含まれる翻字確率をEMアルゴリズムにより更新する、としても良い。
前記更新部は、前記更新による前記尤度関数の期待値の変化量が、所定の大きさ未満となるまで前記更新を繰り返す、としても良い。
前記任意の起源言語における綴りからなる第1文字列と、前記所定の目的言語における綴りからなる第2文字列と、を受け付ける受付部、
前記K個の翻字表のそれぞれを用いることにより前記受け付けられた第1文字列が前記受け付けられた第2文字列に翻字される翻字確率を求め、当該求められた翻字確率を出力する確率出力部、
をさらに備える、としても良い。
前記任意の起源言語における綴りからなる第1文字列を受け付ける受付部、
前記K個の翻字表のそれぞれを用いることにより前記受け付けられた第1文字列が翻字されうる前記所定の目的言語における候補文字列と、当該第1文字列が当該候補文字列に翻字される翻字確率を求め、当該求められた翻字確率が高い順に当該求められた候補文字列を出力する候補出力部、
をさらに備える、としても良い。
前記受付部は、前記訓練集合に含まれていない第1文字列を受け付ける、としても良い。
コンピュータを、
任意の起源言語における綴りからなる原綴り文字列と、当該原綴り文字列を所定の目的言語に翻字した綴りからなる目的綴り文字列と、の翻字対を複数含む訓練集合から、前記原綴り文字列を構成する原断片と、前記目的綴り文字列を構成する翻字断片と、翻字のために当該原断片が当該翻字断片に書き換えられる書換確率との組を複数含む、互いに異なるK個の起源言語に対応したK個の書換表と、前記訓練集合に含まれる翻字対を複数含む、前記K個の起源言語に対応したK個の翻字表とを生成する生成部、
前記訓練集合に複数含まれる翻字対のそれぞれに対して、前記K個の書換表に含まれる書換確率のそれぞれを用いることにより、当該翻字対の原綴り文字列が当該書換表に対応した起源言語を起源とする場合に当該翻字対の目的綴り文字列に翻字される翻字確率を算出してから、当該翻字確率を当該翻字対と対応付けて当該起源言語に対応した翻字表に保存した後に、前記訓練集合が得られた場合における前記K個の翻字表の尤もらしさを表す尤度を算出する尤度関数の期待値であって、当該翻字確率を用いて算出される値を最大化するように、前記K個の書換表のそれぞれに含まれる書換確率を更新した後に、前記翻字確率の算出と前記書換確率の更新とを繰り返す更新部、
として機能させることを特徴とする。
コンピュータを、
任意の起源言語における綴りからなる原綴り文字列と、当該原綴り文字列を所定の目的言語に翻字した綴りからなる目的綴り文字列と、の翻字対を複数含む訓練集合から、前記原綴り文字列を構成する原断片と、前記目的綴り文字列を構成する翻字断片と、翻字のために当該原断片が当該翻字断片に書き換えられる書換確率との組を複数含む、互いに異なるK個の起源言語に対応したK個の書換表と、前記訓練集合に含まれる翻字対を複数含む、前記K個の起源言語に対応したK個の翻字表とを生成する生成部、
前記訓練集合に複数含まれる翻字対のそれぞれに対して、前記K個の書換表に含まれる書換確率のそれぞれを用いることにより、当該翻字対の原綴り文字列が当該書換表に対応した起源言語を起源とする場合に当該翻字対の目的綴り文字列に翻字される翻字確率を算出してから、当該翻字確率を当該翻字対と対応付けて当該起源言語に対応した翻字表に保存した後に、前記訓練集合が得られた場合における前記K個の翻字表の尤もらしさを表す尤度を算出する尤度関数の期待値であって、当該翻字確率を用いて算出される値を最大化するように、前記K個の書換表のそれぞれに含まれる書換確率を更新した後に、前記翻字確率の算出と前記書換確率の更新とを繰り返す更新部、
として機能させることを特徴とする翻字処理プログラムを記録している。
生成部及び更新部とを備える翻字処理装置が実行する方法であって、
前記生成部が、任意の起源言語における綴りからなる原綴り文字列と、当該原綴り文字列を所定の目的言語に翻字した綴りからなる目的綴り文字列と、の翻字対を複数含む訓練集合から、前記原綴り文字列を構成する原断片と、前記目的綴り文字列を構成する翻字断片と、翻字のために当該原断片が当該翻字断片に書き換えられる書換確率との組を複数含む、互いに異なるK個の起源言語に対応したK個の書換表と、前記訓練集合に含まれる翻字対を複数含む、前記K個の起源言語に対応したK個の翻字表とを生成する生成ステップ、
前記更新部が、前記訓練集合に複数含まれる翻字対のそれぞれに対して、前記K個の書換表に含まれる書換確率のそれぞれを用いることにより、当該翻字対の原綴り文字列が当該書換表に対応した起源言語を起源とする場合に当該翻字対の目的綴り文字列に翻字される翻字確率を算出してから、当該翻字確率を当該翻字対と対応付けて当該起源言語に対応した翻字表に保存した後に、前記訓練集合が得られた場合における前記K個の翻字表の尤もらしさを表す尤度を算出する尤度関数の期待値であって、当該翻字確率を用いて算出される値を最大化するように、前記K個の書換表のそれぞれに含まれる書換確率を更新した後に、前記翻字確率の算出と前記書換確率の更新とを繰り返す更新ステップ、
を有することを特徴とする。
図10の第1生成処理を開始すると、第1生成部105aは、図6Aの基礎翻字テーブルを参照することで訓練集合に含まれる翻字対を取得した後に、取得した翻字対を用いて、図9の書換回数テーブルを生成する(ステップS51)。具体的には、第1生成部105aは、訓練集合から全ての翻字対を取り出す。次に、第1生成部105aは、取り出した全ての翻字対に対して、当該翻字対を構成する原綴り文字列snの先頭に特殊文字「^」を追加し、末尾に特殊文字「$」を追加する。その後、第1生成部105aは、特殊文字が追加された文字列snを分割することで原断片αを生成する。
図11の第2生成処理を開始すると、第2生成部105bは、図8Aの基礎書換テーブルを読み込む(ステップS61)。その後、第2生成部105bは、読み込んだ基礎書換テーブルをK個コピーすることで、第1書換テーブルから第K書換テーブルまでのK個の書換テーブルを生成する(ステップS62)。具体的には、第2生成部105bは、図8Aの基礎書換テーブルに保存された断片対を識別する番号jを表す情報と、当該断片対を構成する原断片αを表す情報と、当該断片対を構成する翻字断片βを表す情報と、書換確率P(α→β|z=1)を表す情報と、を対応付けて保存された第1書換テーブルを生成する。この書換確率P(α→β|z=1)は、当該原断片αを含む原綴り文字列snの起源言語が第1言語である条件の下で当該原断片αが翻字断片βに書き換えられる確率をいう。同様に、第2生成部105bは、書換確率P(α→β|z=2)を表す情報を保存された第2書換テーブルと、書換確率P(α→β|z=3)を表す情報を保存された第3書換テーブルと、を生成する。
図11のステップS62において、図8Bの第1書換テーブルに保存された断片対番号「5」で識別される断片対を構成する原断片「get$」が翻字断片「ジェ$」に書き換えられる書換確率P(α→β|z=1)が、他の書換確率P(α→β|z=2)及びP(α→β|z=3)よりも高い値に設定されるとする。この場合、原断片「get$」を含む原綴り文字列s2「xxaget」が翻字断片「ジェ」を含む目的綴り文字列t2「△アジェ」に翻字される翻字確率P(t2|s2,z=1)は、上記の式(3)を用いて算出される。このため、図12Aに示すように、翻字確率P(t2|s2,z=1)は、他の翻字確率P(t2|s2,z=2)及びP(t2|s2,z=3)よりも高くなる。
次に、本実施例の変形例1について説明を行う。
本実施例では、図5の生成部105が、図10のステップS53において、翻字確率Pαβ(sn|tn)を表す情報が保存された図6Aの基礎翻字テーブルを生成してから、図11のステップS64において、当該基礎翻字テーブルに基づいて第1翻字テーブルから第3翻字テーブルを生成するとして説明した。また、本実施例では、図5の更新部107が、図7のステップS37(つまり、Eステップ)の初回の実行時において、第1Eステップを実行しないとして説明した。尚、第1Eステップは、第1書換テーブルから第3書換テーブルを用いたαβ法を実行することで、翻字確率P(tn|sn,z=k)を算出するステップである。
次に、本実施例の変形例2について説明を行う。
変形例2において、図1の端末装置200は、図2Aの入力画面FI1ではなく、図13Aに示すような入力画面FI2を表示することで、任意の起源言語における綴りからなる第1文字列を入力するように促す表示を行う。次に、端末装置200は、ユーザの操作に応じた各種の信号を翻字処理システム1に入力し、入力された信号に基づいて第1文字列を生成し、生成した第1文字列を表す情報を翻字処理装置100へ送信する。その後、端末装置200は、翻字処理装置100から返信された情報を受信する。次に、端末装置200は、図2Bの結果表示画面FO1ではなく、図13Bに示すような結果表示画面FO2を表示する。その後、端末装置200は、受信された情報に基づいて、第1文字列と、当該第1文字列が翻字されうる上記所定の目的言語における候補文字列と、当該第1文字列が当該候補文字列に翻字される確率と、を、結果表示画面FO2上に、当該確率が高い順に一覧表示する。尚、端末装置200は、候補文字列を当該確率が低い順に一覧表示しても良い。また、端末装置200は、当該確率が最も高い第1文字列と候補文字列と当該確率とを1つだけ表示しても良い。さらに、端末装置200は、最も高い確率が所定の閾値を超えている場合に限り、当該最高の確率と1文字列と候補文字列と当該確率とを1つだけ表示しても良い。
次に、本実施例の変形例3について説明を行う。
変形例3の翻字処理システム1は、図1の外国語DBサーバ302を有さず、図16に示すように検索エンジン400を有する。
検索エンジン400は、検索キーワードを表す情報と、当該検索キーワードを含む内容が記載されたWeb(World Wide Web)頁のURLを表す情報と、当該Web頁に記載された内容の内で、当該キーワードを含む部分であるスニペットを表す情報と、を対応付けて記憶している。検索エンジン400は、翻字処理装置100から検索キーワードを表す情報と、検索を求めるリクエストと、を受信すると、受信した検索キーワードを表す情報に対応付けて記憶するスニペットを表す情報を複数検索する。その後、検索エンジン400は、図17に示すようなスニペットSP1及びSP2を含む検索結果画面FSを表す情報を翻字処理装置100へ返信する。
100:翻字処理装置
100a:CPU
100b:ROM
100c:RAM
100d:ハードディスク
100e:メディアコントローラ
100f:LANカード
100g:ビデオカード
100h:LCD
100i:キーボード
100j:スピーカ
100k:マウス
101:訓練集合生成部
102:情報取得部
103:情報記憶部
104:受信部
105:生成部
105a:第1生成部
105b:第2生成部
106:初期化部
107:更新部
107a:期待値算出部
107b:判定部
107c:最大化部
108:確率出力部
109:候補出力部
200:端末装置
301:日本語DBサーバ
302:外国語DBサーバ
400:検索エンジン
Claims (11)
- 任意の起源言語における綴りからなる原綴り文字列と、当該原綴り文字列を所定の目的言語に翻字した綴りからなる目的綴り文字列と、の翻字対を複数含む訓練集合から、前記原綴り文字列を構成する原断片と、前記目的綴り文字列を構成する翻字断片と、翻字のために当該原断片が当該翻字断片に書き換えられる書換確率との組を複数含む、互いに異なるK個の起源言語に対応したK個の書換表と、前記訓練集合に含まれる翻字対を複数含む、前記K個の起源言語に対応したK個の翻字表とを生成する生成部、
前記訓練集合に複数含まれる翻字対のそれぞれに対して、前記K個の書換表に含まれる書換確率のそれぞれを用いることにより、当該翻字対の原綴り文字列が当該書換表に対応した起源言語を起源とする場合に当該翻字対の目的綴り文字列に翻字される翻字確率を算出してから、当該翻字確率を当該翻字対と対応付けて当該起源言語に対応した翻字表に保存した後に、前記訓練集合が得られた場合における前記K個の翻字表の尤もらしさを表す尤度を算出する尤度関数の期待値であって、当該翻字確率を用いて算出される値を最大化するように、前記K個の書換表のそれぞれに含まれる書換確率を更新した後に、前記翻字確率の算出と前記書換確率の更新とを繰り返す更新部、
を備えることを特徴とする翻字処理装置。 - 請求項1に記載の翻字処理装置であって、
前記K個の翻字表それぞれに対応した起源言語を起源とする原綴り文字列が、前記訓練集合に含まれる複数の原綴り文字列に占める割合をそれぞれ示す、前記K個の翻字表の重みパラメタを初期化する初期化部、を更に備え、
前記更新部は、前記訓練集合に複数含まれる翻字対のそれぞれに対して、前記K個の翻字表に含まれる当該翻字確率の前記重みパラメタによる重み付き平均に基づいて、前記原綴り文字列の起源言語が当該翻字表に対応した起源言語である起源確率をそれぞれ算出した後に、当該起源確率をさらに用いて算出される前記尤度関数の期待値を最大化するように、前記K個の重みパラメタを更新した後に、前記起源確率の算出と前記重みパラメタの更新とを繰り返す、
ことを特徴とする翻字処理装置。 - 請求項1又は2に記載の翻字処理装置であって、
前記生成部は、1個の前記書換表を生成した後に、前記生成された1個の書換表に含まれる書換確率を、前記K個の言語毎に変化させることにより、前記K個の言語に対応したK個の書換表を生成する、
ことを特徴とする翻字処理装置。 - 請求項1又は2に記載の翻字処理装置であって、
前記生成部は、1個の前記書換表を用いたαβ法により1個の前記翻字表を生成した後に、前記生成された1個の翻字表に含まれる翻字確率を変化させることにより、前記K個の言語に対応したK個の翻字表を生成し、
前記更新部は、前記K個の翻字表のそれぞれに含まれる翻字確率をEMアルゴリズムにより更新する、
ことを特徴とする翻字処理装置。 - 請求項1から4のいずれか一項に記載の翻字処理装置であって、
前記更新部は、前記更新による前記尤度関数の期待値の変化量が、所定の大きさ未満となるまで前記更新を繰り返す、
ことを特徴とする翻字処理装置。 - 請求項1から5のいずれか一項に記載の翻字処理装置であって、
前記任意の起源言語における綴りからなる第1文字列と、前記所定の目的言語における綴りからなる第2文字列と、を受け付ける受付部、
前記K個の翻字表のそれぞれを用いることにより前記受け付けられた第1文字列が前記受け付けられた第2文字列に翻字される翻字確率を求め、当該求められた翻字確率を出力する確率出力部、
をさらに備えることを特徴とする翻字処理装置。 - 請求項1から5のいずれか一項に記載の翻字処理装置であって、
前記任意の起源言語における綴りからなる第1文字列を受け付ける受付部、
前記K個の翻字表のそれぞれを用いることにより前記受け付けられた第1文字列が翻字されうる前記所定の目的言語における候補文字列と、当該第1文字列が当該候補文字列に翻字される翻字確率を求め、当該求められた翻字確率が高い順に当該求められた候補文字列を出力する候補出力部、
をさらに備えることを特徴とする翻字処理装置。 - 請求項6又は7に記載の翻字処理装置であって、
前記受付部は、前記訓練集合に含まれていない第1文字列を受け付ける、
ことを特徴とする翻字処理装置。 - コンピュータを、
任意の起源言語における綴りからなる原綴り文字列と、当該原綴り文字列を所定の目的言語に翻字した綴りからなる目的綴り文字列と、の翻字対を複数含む訓練集合から、前記原綴り文字列を構成する原断片と、前記目的綴り文字列を構成する翻字断片と、翻字のために当該原断片が当該翻字断片に書き換えられる書換確率との組を複数含む、互いに異なるK個の起源言語に対応したK個の書換表と、前記訓練集合に含まれる翻字対を複数含む、前記K個の起源言語に対応したK個の翻字表とを生成する生成部、
前記訓練集合に複数含まれる翻字対のそれぞれに対して、前記K個の書換表に含まれる書換確率のそれぞれを用いることにより、当該翻字対の原綴り文字列が当該書換表に対応した起源言語を起源とする場合に当該翻字対の目的綴り文字列に翻字される翻字確率を算出してから、当該翻字確率を当該翻字対と対応付けて当該起源言語に対応した翻字表に保存した後に、前記訓練集合が得られた場合における前記K個の翻字表の尤もらしさを表す尤度を算出する尤度関数の期待値であって、当該翻字確率を用いて算出される値を最大化するように、前記K個の書換表のそれぞれに含まれる書換確率を更新した後に、前記翻字確率の算出と前記書換確率の更新とを繰り返す更新部、
として機能させることを特徴とする翻字処理プログラム。 - コンピュータを、
任意の起源言語における綴りからなる原綴り文字列と、当該原綴り文字列を所定の目的言語に翻字した綴りからなる目的綴り文字列と、の翻字対を複数含む訓練集合から、前記原綴り文字列を構成する原断片と、前記目的綴り文字列を構成する翻字断片と、翻字のために当該原断片が当該翻字断片に書き換えられる書換確率との組を複数含む、互いに異なるK個の起源言語に対応したK個の書換表と、前記訓練集合に含まれる翻字対を複数含む、前記K個の起源言語に対応したK個の翻字表とを生成する生成部、
前記訓練集合に複数含まれる翻字対のそれぞれに対して、前記K個の書換表に含まれる書換確率のそれぞれを用いることにより、当該翻字対の原綴り文字列が当該書換表に対応した起源言語を起源とする場合に当該翻字対の目的綴り文字列に翻字される翻字確率を算出してから、当該翻字確率を当該翻字対と対応付けて当該起源言語に対応した翻字表に保存した後に、前記訓練集合が得られた場合における前記K個の翻字表の尤もらしさを表す尤度を算出する尤度関数の期待値であって、当該翻字確率を用いて算出される値を最大化するように、前記K個の書換表のそれぞれに含まれる書換確率を更新した後に、前記翻字確率の算出と前記書換確率の更新とを繰り返す更新部、
として機能させることを特徴とする翻字処理プログラムを記録したコンピュータ読み取り可能な記録媒体。 - 生成部及び更新部とを備える翻字処理装置が実行する方法であって、
前記生成部が、任意の起源言語における綴りからなる原綴り文字列と、当該原綴り文字列を所定の目的言語に翻字した綴りからなる目的綴り文字列と、の翻字対を複数含む訓練集合から、前記原綴り文字列を構成する原断片と、前記目的綴り文字列を構成する翻字断片と、翻字のために当該原断片が当該翻字断片に書き換えられる書換確率との組を複数含む、互いに異なるK個の起源言語に対応したK個の書換表と、前記訓練集合に含まれる翻字対を複数含む、前記K個の起源言語に対応したK個の翻字表とを生成する生成ステップ、
前記更新部が、前記訓練集合に複数含まれる翻字対のそれぞれに対して、前記K個の書換表に含まれる書換確率のそれぞれを用いることにより、当該翻字対の原綴り文字列が当該書換表に対応した起源言語を起源とする場合に当該翻字対の目的綴り文字列に翻字される翻字確率を算出してから、当該翻字確率を当該翻字対と対応付けて当該起源言語に対応した翻字表に保存した後に、前記訓練集合が得られた場合における前記K個の翻字表の尤もらしさを表す尤度を算出する尤度関数の期待値であって、当該翻字確率を用いて算出される値を最大化するように、前記K個の書換表のそれぞれに含まれる書換確率を更新した後に、前記翻字確率の算出と前記書換確率の更新とを繰り返す更新ステップ、
を有することを特徴とする翻字処理方法。
Priority Applications (6)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US13/825,899 US9323744B2 (en) | 2011-03-04 | 2012-02-28 | Transliteration device, transliteration program, computer-readable recording medium on which transliteration program is recorded, and transliteration |
| KR1020137007988A KR101326354B1 (ko) | 2011-03-04 | 2012-02-28 | 문자 변환 처리 장치, 기록 매체 및 방법 |
| CA2813218A CA2813218C (en) | 2011-03-04 | 2012-02-28 | Transliteration device, transliteration program, computer-readable recording medium on which transliteration program is recorded, and transliteration method |
| EP12755516.7A EP2672395A4 (en) | 2011-03-04 | 2012-02-28 | TRANSLITERATION DEVICE AND PROGRAM, RECORDING MEDIA AND METHOD THEREFOR |
| CN201280003171.9A CN103140849B (zh) | 2011-03-04 | 2012-02-28 | 音译处理装置及方法 |
| BR112013007127A BR112013007127A2 (pt) | 2011-03-04 | 2012-02-28 | dispositivo e método de transliteração, e, mídia de gravação legível por computador |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2011048384A JP5090547B2 (ja) | 2011-03-04 | 2011-03-04 | 翻字処理装置、翻字処理プログラム、翻字処理プログラムを記録したコンピュータ読み取り可能な記録媒体、及び翻字処理方法 |
| JP2011-048384 | 2011-03-04 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2012121063A1 true WO2012121063A1 (ja) | 2012-09-13 |
Family
ID=46798031
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/JP2012/054956 Ceased WO2012121063A1 (ja) | 2011-03-04 | 2012-02-28 | 翻字処理装置、プログラム、記録媒体、及び方法 |
Country Status (9)
| Country | Link |
|---|---|
| US (1) | US9323744B2 (ja) |
| EP (1) | EP2672395A4 (ja) |
| JP (1) | JP5090547B2 (ja) |
| KR (1) | KR101326354B1 (ja) |
| CN (1) | CN103140849B (ja) |
| BR (1) | BR112013007127A2 (ja) |
| CA (1) | CA2813218C (ja) |
| TW (1) | TWI465941B (ja) |
| WO (1) | WO2012121063A1 (ja) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017002199A1 (ja) * | 2015-06-30 | 2017-01-05 | 楽天株式会社 | 翻字処理装置、翻字処理方法、翻字処理プログラム、及び情報処理装置 |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5825639B2 (ja) * | 2012-07-20 | 2015-12-02 | 日本電信電話株式会社 | 記号列対応付け装置、記号列変換モデル学習装置、記号列変換装置、方法、及びプログラム |
| US9176936B2 (en) * | 2012-09-28 | 2015-11-03 | International Business Machines Corporation | Transliteration pair matching |
| JP6044996B2 (ja) * | 2013-07-18 | 2016-12-14 | 日本電信電話株式会社 | 文字列対応付け装置、方法、及びプログラム |
| KR101609184B1 (ko) * | 2014-05-27 | 2016-04-06 | 네이버 주식회사 | 사전 기능을 제공하는 방법과 시스템, 그리고 기록 매체 및 파일 배포 시스템 |
| CN105786802B (zh) * | 2014-12-26 | 2019-04-12 | 广州爱九游信息技术有限公司 | 一种外语的音译方法及装置 |
| US10225555B2 (en) | 2015-05-19 | 2019-03-05 | Mediatek Inc. | Method and apparatus for multi-table based context adaptive binary arithmetic coding |
| RU2632137C2 (ru) * | 2015-06-30 | 2017-10-02 | Общество С Ограниченной Ответственностью "Яндекс" | Способ и сервер транскрипции лексической единицы из первого алфавита во второй алфавит |
| KR101917648B1 (ko) * | 2016-09-08 | 2018-11-13 | 주식회사 하이퍼커넥트 | 단말 및 그 제어 방법 |
| KR102197227B1 (ko) * | 2019-05-23 | 2020-12-31 | 한국과학기술원 | 문자 변환 방법 및 이를 수행하는 장치들 |
| CN110728156B (zh) * | 2019-12-19 | 2020-07-10 | 北京百度网讯科技有限公司 | 翻译方法、装置、电子设备及可读存储介质 |
| JP7692715B2 (ja) * | 2021-03-26 | 2025-06-16 | 一般財団法人日本特許情報機構 | 読み方推定装置、読み方推定方法、読み方推定プログラム、および記録媒体 |
| US12298972B2 (en) | 2021-10-27 | 2025-05-13 | Bank Of America Corporation | System and method for recursive transliteration of machine interpretable languages |
| JP7829861B2 (ja) * | 2022-06-16 | 2026-03-16 | 国立研究開発法人情報通信研究機構 | 機械翻訳用訓練データ生成方法、機械翻訳処理用の学習可能モデルの作成方法、機械翻訳処理方法、および、機械翻訳用訓練データ生成装置 |
| US12360990B2 (en) * | 2022-11-03 | 2025-07-15 | Bank Of America Corporation | Transliteration of machine interpretable languages for enhanced compaction |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6272464B1 (en) * | 2000-03-27 | 2001-08-07 | Lucent Technologies Inc. | Method and apparatus for assembling a prediction list of name pronunciation variations for use during speech recognition |
| JP2005092682A (ja) * | 2003-09-19 | 2005-04-07 | Nippon Hoso Kyokai <Nhk> | 翻字装置、及び翻字プログラム |
| JP2007156545A (ja) * | 2005-11-30 | 2007-06-21 | Nippon Telegr & Teleph Corp <Ntt> | 記号列変換方法、単語翻訳方法、その装置およびそのプログラム並びに記録媒体 |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7369986B2 (en) * | 2003-08-21 | 2008-05-06 | International Business Machines Corporation | Method, apparatus, and program for transliteration of documents in various Indian languages |
| US7310605B2 (en) * | 2003-11-25 | 2007-12-18 | International Business Machines Corporation | Method and apparatus to transliterate text using a portable device |
| US8548791B2 (en) * | 2007-08-29 | 2013-10-01 | Microsoft Corporation | Validation of the consistency of automatic terminology translation |
| US7983903B2 (en) * | 2007-09-07 | 2011-07-19 | Microsoft Corporation | Mining bilingual dictionaries from monolingual web pages |
| WO2009049049A1 (en) * | 2007-10-09 | 2009-04-16 | Language Analytics Llc | Method and system for adaptive transliteration |
| CN101593173B (zh) * | 2008-05-28 | 2011-08-10 | 中国科学院自动化研究所 | 一种汉英反向音译方法及装置 |
| US8521761B2 (en) * | 2008-07-18 | 2013-08-27 | Google Inc. | Transliteration for query expansion |
| US8275600B2 (en) * | 2008-10-10 | 2012-09-25 | Google Inc. | Machine learning for transliteration |
| US8306806B2 (en) * | 2008-12-02 | 2012-11-06 | Microsoft Corporation | Adaptive web mining of bilingual lexicon |
| US20110218796A1 (en) * | 2010-03-05 | 2011-09-08 | Microsoft Corporation | Transliteration using indicator and hybrid generative features |
| US8326600B2 (en) * | 2010-08-11 | 2012-12-04 | Google Inc. | Evaluating and modifying transliteration rules |
-
2011
- 2011-03-04 JP JP2011048384A patent/JP5090547B2/ja active Active
-
2012
- 2012-02-28 EP EP12755516.7A patent/EP2672395A4/en not_active Withdrawn
- 2012-02-28 BR BR112013007127A patent/BR112013007127A2/pt not_active Application Discontinuation
- 2012-02-28 WO PCT/JP2012/054956 patent/WO2012121063A1/ja not_active Ceased
- 2012-02-28 KR KR1020137007988A patent/KR101326354B1/ko active Active
- 2012-02-28 CN CN201280003171.9A patent/CN103140849B/zh active Active
- 2012-02-28 CA CA2813218A patent/CA2813218C/en active Active
- 2012-02-28 US US13/825,899 patent/US9323744B2/en active Active
- 2012-03-02 TW TW101107023A patent/TWI465941B/zh active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6272464B1 (en) * | 2000-03-27 | 2001-08-07 | Lucent Technologies Inc. | Method and apparatus for assembling a prediction list of name pronunciation variations for use during speech recognition |
| JP2005092682A (ja) * | 2003-09-19 | 2005-04-07 | Nippon Hoso Kyokai <Nhk> | 翻字装置、及び翻字プログラム |
| JP2007156545A (ja) * | 2005-11-30 | 2007-06-21 | Nippon Telegr & Teleph Corp <Ntt> | 記号列変換方法、単語翻訳方法、その装置およびそのプログラム並びに記録媒体 |
Non-Patent Citations (4)
| Title |
|---|
| ERIC BRILL, AN IMPROVED ERROR MODEL FOR NOISY CHANNEL SPELLING CORRECTION, 2000, pages 286 - 293 |
| HAIZHOU LI, SEMANTIC TRANSLITERATION OF PERSONAL NAMES, June 2007 (2007-06-01), pages 120 - 127 |
| HAJIME MOCHIZUKI ET AL.: "Automatic Transliteration and Back-transliteration System for Languages That UseNon-Roman Alphabets in Bibliographic Citation Database", TRANSACTIONS OF INFORMATION PROCESSING SOCIETY OF JAPAN, vol. 46, 15 December 2005 (2005-12-15), pages 50 - 58, XP055125971 * |
| See also references of EP2672395A4 |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017002199A1 (ja) * | 2015-06-30 | 2017-01-05 | 楽天株式会社 | 翻字処理装置、翻字処理方法、翻字処理プログラム、及び情報処理装置 |
| JP6067952B1 (ja) * | 2015-06-30 | 2017-01-25 | 楽天株式会社 | 翻字処理装置、翻字処理方法、翻字処理プログラム、及び情報処理装置 |
| US10185710B2 (en) | 2015-06-30 | 2019-01-22 | Rakuten, Inc. | Transliteration apparatus, transliteration method, transliteration program, and information processing apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| CA2813218A1 (en) | 2012-09-13 |
| KR20130038959A (ko) | 2013-04-18 |
| TWI465941B (zh) | 2014-12-21 |
| US20130246042A1 (en) | 2013-09-19 |
| KR101326354B1 (ko) | 2013-11-11 |
| CA2813218C (en) | 2014-04-15 |
| TW201250498A (en) | 2012-12-16 |
| JP5090547B2 (ja) | 2012-12-05 |
| US9323744B2 (en) | 2016-04-26 |
| JP2012185679A (ja) | 2012-09-27 |
| CN103140849B (zh) | 2015-01-14 |
| BR112013007127A2 (pt) | 2016-06-14 |
| CN103140849A (zh) | 2013-06-05 |
| EP2672395A1 (en) | 2013-12-11 |
| EP2672395A4 (en) | 2015-07-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5090547B2 (ja) | 翻字処理装置、翻字処理プログラム、翻字処理プログラムを記録したコンピュータ読み取り可能な記録媒体、及び翻字処理方法 | |
| US8612206B2 (en) | Transliterating semitic languages including diacritics | |
| US9262403B2 (en) | Dynamic generation of auto-suggest dictionary for natural language translation | |
| US8626486B2 (en) | Automatic spelling correction for machine translation | |
| JP5379138B2 (ja) | 領域辞書の作成 | |
| US11468336B2 (en) | Systems, devices, and methods for improved affix-based domain name suggestion | |
| JP6705318B2 (ja) | 対訳辞書作成装置、対訳辞書作成方法、及び対訳辞書作成プログラム | |
| CN101826072A (zh) | 计算机辅助自然语言翻译 | |
| WO2003065245A1 (en) | Translating method, translated sentence outputting method, recording medium, program, and computer device | |
| JP2009026100A (ja) | 処理対象として適切なテキストを選択する技術 | |
| US20190303437A1 (en) | Status reporting with natural language processing risk assessment | |
| US10650195B2 (en) | Translated-clause generating method, translated-clause generating apparatus, and recording medium | |
| JP7591212B1 (ja) | 情報処理装置、情報処理方法、およびプログラム | |
| JP2018147205A (ja) | 情報処理装置及び情報処理プログラム | |
| JP2019016074A (ja) | オントロジー生成装置、オントロジー生成プログラム及びオントロジー生成方法 | |
| JP6584361B2 (ja) | キーワード抽出装置、キーワード抽出方法及びキーワード抽出プログラム | |
| US20250045315A1 (en) | Information processing device, information processing system, information processing method, and storage medium | |
| JP7483085B1 (ja) | 情報処理システム、情報処理装置、情報処理方法、およびプログラム | |
| WO2009144890A1 (ja) | 翻訳前換言規則生成システム | |
| JP5378109B2 (ja) | タスクモデル生成装置およびタスクモデル生成方法 | |
| JP4203102B2 (ja) | 中国語解析装置、中国語解析方法および中国語解析プログラム | |
| Cocu | questionable Use of Anglicisms in Romanian Computerese | |
| JP2024034877A (ja) | 入力データ作成支援装置、及び入力データ作成支援方法 | |
| JP2025182956A (ja) | 秘匿判定装置、秘匿判定方法、及びプログラム | |
| CN116257605A (zh) | 用于查找自然语言与计算机语言之间的关联的方法和系统 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| WWE | Wipo information: entry into national phase |
Ref document number: 201280003171.9 Country of ref document: CN |
|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 12755516 Country of ref document: EP Kind code of ref document: A1 |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 2012755516 Country of ref document: EP |
|
| ENP | Entry into the national phase |
Ref document number: 20137007988 Country of ref document: KR Kind code of ref document: A Ref document number: 2813218 Country of ref document: CA |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 13825899 Country of ref document: US |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| REG | Reference to national code |
Ref country code: BR Ref legal event code: B01A Ref document number: 112013007127 Country of ref document: BR |
|
| ENP | Entry into the national phase |
Ref document number: 112013007127 Country of ref document: BR Kind code of ref document: A2 Effective date: 20130326 |





