Using a neural net to determine the language in which a text is written

C. Lyon, C. Matthews

    Research output: Book/ReportOther report

    49 Downloads (Pure)

    Abstract

    There are statistical patterns of letter sequences in natural language, and different languages have different characteristic patterns. This effect can be used to determine in which language a text is written. The patterns are captured with a single layer, feed forward neural net trained in supervised mode. The sequential dependencies of letters are modelled by taking adjacent letter pairs and letter triples. Training and test data are converted to sets of these tuples, which are the basic elements classified by the network. This approach is supported by information theoretic results on the entropy of letter sequences for English. The architecture of the network used is shown to be appropriate for data with the characteristics of natural language letter sequences. For 3 languages over 99% of test strings are correct. For 4 languages, including Dutch and German which are similar, over 92% are correct.
    Original languageEnglish
    PublisherUniversity of Hertfordshire
    Publication statusPublished - 1995

    Publication series

    NameUH Computer Science Technical Report
    PublisherUniversity of Hertfordshire
    Volume212

    Fingerprint

    Dive into the research topics of 'Using a neural net to determine the language in which a text is written'. Together they form a unique fingerprint.

    Cite this