Natural Language Processing – how does AI communicate?

From now on, intelligent systems should also be linguistically proficient so that they can interact with us humans. But how can our natural language be processed in a way that it becomes tangible and usable for an artificial intelligence? This question is addressed by a branch of machine learning, the so-called Natural Language Processing (NLP).

Natural Language Processing pursues one goal: An artificial intelligence should be taught a language in a way that it can understand and use texts and language syntactically and semantically correct. The whole thing is relatively demanding, because as you know from your own experience, translating and understanding requires a high level of language understanding.

But how do NLP procedures work?

In order to understand the NLP technics, it is advisable to first get a rough understanding of what the term “language” encompasses. In general, we refer to the written as well as the spoken word as language. But non-verbal communication, such as facial expressions and gestures, also has a great influence on the meaning of words. This already gives an idea of the basic problem of NLP: It is not enough to train the machine with a certain number of words. Rather, the AI must also be taught the meaning of each word in its respective context. And this is the real crux of the matter – to present a given context in a way that the computer can understand it.

Even before the invention of NLP, researchers have been working on similar tasks. For this purpose, they usually looked at prefixes, suffixes and the respective word stem and then started a segment analysis. Today’s NLP is normally based on deep learning methods. The mostly vector-based procedures include the former systematics.

The basic idea of the vector-based techniques is to span single sense units – the words – in a multidimensional vector space. As a result words with similar meanings are closer together.

The basic idea behind vector-based techniques is the so-called one-hot vector: each word is assigned a complex, multidimensional vector. This gives each word a fixed position within a data space. If, for example, a text consisting of 1,000 words is taken as a basis, a separate vector is created for each word. The first word is given a vector with the number 1 and then 999 times zero. The vector of the second word contains the number zero as its first component, the number one as its second component and then 998 times the number zero. The following words behave analogously, in that the position of the number one is always shifted one position to the right or downwards, depending on how the vector is represented.

Word 1: Apple (1/0/0/0/0/0/…)

Word 2: Pear (0/1/0/0/0/0/…)

How linguistically proficient are intelligent systems today?

AIs are currently able to separate several superimposed sound tracks spoken by humans and understand the meaning. But there are also some hurdles that still need to be overcome. Everyone of us knows the phenomenon: If you start learning a foreign language the beginning is always designed so that a good foundation can be established relatively quickly. For becoming a fluent and grammatically perfect speaker,  the rhetorical fine-tuning requires a far greater effort. This behaviour is also transferable to AI.

Photo by Raphael Schaller on Unsplash