Introduction
I know like me, as a child, you probably daydreamed about how amazing life would be like to have a walking and talking robot. I used to imagine ordering my robot to make my bed, clean up my room and most definitely do all of my homework! I would have time to spend on the real things that were of much greater importance. Little did I know back then that it is a complicated task to communicate with a machine.
Communication is often a two-way process. It is necessary to not only express our thoughts impeccably but to comprehend others’ words without our biases. It is a bonus if we can predict others’ words! It is clear that communication is critical, and to communicate well is a very desirable skill. According to a popular Holmes Report on “The Cost of Poor Communications”, the cost of inadequate communication is staggering for businesses. It is evident that companies that have leaders who are highly effective communicators have relevant business models, higher profits and at least 50% higher returns to shareholders. They constantly seek to learn how they can serve their customers and in turn grow their profits.
Diving into The Complexity of Language
So, what makes understanding human language so difficult to understand? It is something to ponder over - we humans are certainly not experts at communicating with each other. How can we communicate with a machine? The challenge is attributed to the dynamism of human language. For example, understanding the query intent is a complex process. If we remove all contextual information, then only the key words remain. This can be quite confusing! How would my robot comprehend my direction “Make my bed!”? Does this mean actually hammering wooden boards to construct a bed structure or does it mean to neatly arrange the bedsheets? This illustrates syntactical complexity in human language.
We must keep in mind that all human languages have evolved over thousands of years of speech patterns. Language is essentially a fluid, living entity that develops with the needs and situations of communities. If we think about Shakespearean English and the English we speak today, we can easily notice the drastic contrast. According to Grammarly, in Shakespearean times (late 1500s – early 1600), the words bandit, lonely, critic, dauntless, dwindle, elbow, green-eyed (to describe jealousy) and lackluster were created. It is interesting to observe the turbulence during that time – resulting in the creation of new vocabulary. Additionally, the tone, sentiment and mentality of society was very different – all providing a distinctive filter on communication.
The Challenge of Human-Machine Understanding
As humans we can use our intuitions and communication experiences to understand even if it something not explicitly stated. In contrast, a machine lacks intuition. However, intuition can be developed with the “experience” of a larger corpus. For example, we have sufficient life experiences to understand what a lion is. A computer cannot comprehend what a lion is as an entity or define its attributes. A computer is exceptional at computing the probability of a lion moving as higher than the probability of a piano moving. Regardless of how many layers of natural language methodologies we implement and the quantity of text, it is impossible to recreate human intuition and experience in a machine.
Our best approach in converging on the communication gap between human and machine is to represent words relative to other words within a corpus. Natural language processing (NLP) is an umbrella technology encompassing everything from text parsing to complex statistical methods used in deep learning. The aim is to enable communication between a machine and humans. NLP methodologies process human language. NLP allows computers to communicate and comprehend by reading, editing, summarizing text and even generating text.
NLP’s Impact on Machine Learning
There are many open-source techniques to aid a machine to understand text. Text embedding techniques, such as Word2vec (from Google) and GloVe (from Stanford), provide a general natural language methodology to cultivate an understanding of words, context, sentiment and intent. Words are represented as a dense vector in a highly dimensional sparse matrix. We build a dense vector so that it is similar to vectors of words that appear in similar contexts. Using these vector representations, it is possible to find how different words in a sentence relate to each other and how they relate collectively. The algorithm goes through each position of a word in the text. The word vectors are iteratively adjusted until this probability is maximized. If a word in a sentence is replaced with a word with a comparable vector representation, we can obtain a similar meaning for the sentence. Word2vec performs wonderfully when we have a large corpus. Word2vec training can be improved by eliminating stopwords from the dataset. Stopwords are high frequency words that add little value to language understanding, and their removal aids in improving model accuracy and training time.
About Cedrus: If you’d like a guided approach for implementing NLP on your road towards Enterprise-AI, Cedrus Digital specializes in the AI transformation journey for companies of all sizes. Come work with our experts to set you on your path in not only NLP, but Conversational AI, Vision, Predictive Analytics, and Knowledge Graphs as well. We can help you brainstorm& prioritize use cases, as well as help with planning, management, and delivery of AI projects. Let’s partner together.
In future blogposts I will dive deeper into the intricacies of text embedding and explore, with some technical rigor, other aspects of NLP required for successful implementation. Businesses have realized that unstructured and semi-structured data need to be mined using NLP rather than rely on outdated manual or template-based techniques. Natural Language Processing is powerful tool that has an immense impact on the business model and serving the customer. Stay tuned!
Swati Sharma, Ph.D. is a Senior AI Solutions Engineer at Cedrus Digital. She teaches and mentors future data scientists and works with clients to create solutions for complex business problems.