This is my report for the CMU LTI colloquium.
Video link: https://www.youtube.com/watch?v=cPWFJE6FKeY
Title: The Words are Alive: Associations with Sentiment, Emotion, Colour, and Music!
Speaker: Saif Mohammad (National Research Council Canada)
Date: September 19, 2014
This presentation deals mainly with the speaker’s work on NLP, specifically sentiment analysis, emotion, color, and music. Most of the work starts from the sentiment lexicon he built using Amazon Mechanical Turk. While his work includes pure text sentiment (emotion) analysis, he has also made effort to sonify and visualize the emotional information of words. This presentation was very interesting to me for several reasons. First, I worked on sentiment analysis in the past, so I have some sense of what have been hot research topics and approaches in this research field. I also wanted to know how the speaker approached and solved the problem. Furthermore, now that I no longer work on sentiment analysis, I wanted to hear state-of-the-art techniques and applications. Second, the speaker’s work not only focused on techniques, but also combined the techniques with art. Image and music make me happy and excited. This aspect of his work impressed me and caused me to think a lot about my future research and thesis.
While most of his work is based on sentiment (emotion) analysis and building of sentiment lexicon, he gave only a brief introduction of sentiment lexicon construction, and then presented its application to sonification and visualization. Only after then did he bring us into several techniques for better sentiment analysis. It was a good strategy because sonification and visualization was the most fun part. I’ll also follow that order in this report.
The speaker utilized crowdsourcing to collect sentiment words. For most work on sentiment analysis, sentiment lexicons are the fundamental resource. It can be said that the performance of sentiment analysis depends mostly on the quality of the sentiment lexicon used. There have been a lot of efforts on building sentiment lexicons. However, because of the variety of domains, there are no generally used lexicons. Rather, classical, small lexicons such as LIWC and General Inquirer are widely used in research for English. These lexicons were compiled by psychologists, and it seems like sentiment lexicons are often asked about their validity in psychological perspectives. Many other languages do not even have a small lexicon. In this case, crowdsourcing is a good way to collect sentiment words.
The speaker and his colleague Hanna Davis tried to generate music from novels automatically. They first divided a novel into sections and measured the degree of emotion of each section on the basis of their sentiment lexicon. If a section contains much emotion, the section was divided into fine-grained segments. Music is then generated for each segment according to the proportions of emotions. They made some building blocks for this; for example, happiness is associated with major keys and sadness with minor keys. Happiness and excitement make the tempo fast. Joy and calm generate a sequence of consonant notes, while excitement, anger, and unpleasantness prefer inharmonics. Music is important for novels as well as movies. For example, a famous Japanese novelist Murakami Haruki strategically associates each of his novels with a music work; the music is mentioned many times in the story and readers can enjoy the novel by reading it and listening to the music in both ways. By listening to the generated music, we can catch and enjoy the overall emotional flow of the novel.
While these rules are intuitive and make sense, there are still many challenges. Most of all, generated music should be listenable and pleasant instead of being a random sequence of notes. I listened to several results of this work on the website, and in that perspective, this work is far from the best. The main problem is that the music has no beautiful melody line. I would suggest that this work take advantage of existing research on music synthesis. I do not know well about that research field, but one straightforward approach could be to use the segments of existing music and assemble them, instead of generating music from scratch. Since this work is based on the researchers’ intuition on the relation between emotion and musical features, it would be worth trying to automatically extract musical features using machine learning techniques. Similarly, another limitation of this work is that some emotions (e.g., “trust”) are not associated with intuitive musical features. According to the results, many novels have “turst” as their highest emotion, which makes it hard to interpret conveying emotions in them.
The speaker and his colleagues also worked on word-color association. For example, “iceberg” is highly associated with white and “vegetation” with green. They again crowdsourced the information and built 24,200 word-color pairs with total 11 colors. Research questions could be whether the associations have high agreement, whether concrete concepts have higher agreement, and so on. Like the music case, visualization is also important especially for marketing purposes. Companies try to associate their brands with certain colors or images in order to make their identity clear as well as to remind people of their brands with certain visual cues. The color of a logo plays a big role in a way to perceive the company; e.g., blue with trustfulness, white with cleanness, etc. Therefore, word-color association is useful and intensely used information for marketers. This work is thus beneficial in the sense that it used a quite large-scale method (crowdsourcing) to obtain that information and investigated the validity of the information. However, the work does not use a data-driven approach, which may yield a result biased by human consciousness.
The speaker’s work on word-music and word-color association was impressive, especially because this is not what I can easily think of. This work does not contain heavy mathematics or fancy machine learning techniques. I am somewhat skewed toward math-intensive research, often missing fun and joyful ideas. This work does not have correct answers, and evaluation is not straightforward. I would say it’s more like Human-Computer Interaction than Language Technologies. Although the goal and usefulness may seem somewhat vague and not organized at the initial stage, follow-up work can build on this and produce useful results afterward. Along this line, I thought about whether this research could be a good thesis topic. People often emphasize the importance of formalizing a new problem and novelty for thesis work. This research is novel; the visualization and sonification of text have been done mainly by humans. In addition, traditional approaches depend on human intuition and small-scale surveys, rather than scientific and scalable methodologies. Computer-aided and data-driven approaches may be able to help experts get deeper insights, counterintuitive information, and statistically reliable information.
Regarding the techniques the speaker used to build sentiment lexicons, they have similarities and differences to my approaches. One similarity is that we both used Point-wise Mutual Information to obtain sentiment scores of general words occurring with seed sentiment words. Their work used Twitter hashtags for seed words, and in one study in 2010, I used emoticons as seed words. It is also similar that we both considered negation. However, while I replaced a negated word by prefixing it with “no_”, they seemed to use different methods but it is not very clear exactly what they did. They tackled the polarity degree problem too, which I did not. This problem is to assign each word a score of polarity intensity. As the speaker pointed out in the presentation, this problem is hard because we humans are bad at assigning absolute real value to words. On the other hand, humans are good at comparing two words, so they formulated the problem as comparing the intensities of two words. I also gave a thought to this problem before, but did not make meaningful progress.