Prezydent Andrzej Dudy: Automatyczna Analiza Spontanicznej Ekspresji Twarzy

For English version click here.

Automatyczna Analiza Spontanicznej Ekspresji Twarzy Prezydenta dr. Andrzej Dudy podczas rozmowy z internautami na Facebook’u 06 Sierpnia 2015.

Przeanalizowane zostało wideo skierowane do polskich internautów przez nowo mianowanego Prezydenta RP na Facebooku. Przemówienie było transmitowane w Internecie w dniu 6 sierpnia 2015. Tylko 2,5 godziny po oficjalnym powołaniu na urząd Prezydenta RP, Andrzej Duda odpowiadał na pytania internautów. Większość analiz w prasie skupia się na tym co zostało powiedziane, ta analiza koncentruje się na tym jak to zostało powiedziane i jakie emocje pokazywał Prezydent Andrzej Duda.

FR_view

Rysunek 1. Wizualizacja analizy FaceReader’a.

Analiza FaceReader: FACS, Rytm Serca, Walencja Afektu / Podniecenie i Emocje Podstawowe

Powyższy Rysunek 1 ilustruje, że FaceReader udostępnia cztery różne metody analizy emocji twarzy. Pierwsza to System Kodwania Ruchów Miminczych. Liczby na twarz – 2, 6, 7 i 12 z odpowiednimi oceny intensywności (ciemniejszy kolor, tym bardziej intensywne ekspresji) – reprezentują Jednostki Akcji (AU) na podstawie Systemu Kodowania Ruchów Mimicznych (Facial Action Coding System; Ekman i in., 2002). Grafika czerwonego serca z liczbą (w tym przypadku 84) wskazuje uderzenia serca na minutę (bpm), który jest miarą częstości akcji serca. Następnie Kołowy Model Afektu (Russel, 1980) ilustruje walencję afektu (przyjemność / nieprzyjemność) i pobudzenie (pobudzenie / brak pobudzenia ). Wykres na dole wskazuje podstawowe emocje (Radość, Smutek, Zlość, Zaskoczenie, Strach, Wstręt; Ekman i wsp. 1969).

Kontekst analizy

W tej automatyzcznej analizie ekspresji emocji twarzy, zostały połączone trzy ciekawe fakty: a) pierwsze oficjalne przemówienie nowego polskiego Prezydenta, który b) rozmawiał  z internautami używając nowej usługi Facebooka oraz c) zastosowanie nowej metody – komputerowej i automatycznej analizy spontanicznej ekspresji twarzy, przez program zwany FaceReader (2015).

Wideo-chat trwał 25 minut, 10 sekund, czyli ponad 45’000 klatek. Przeanalizowane zostało to wideo klatka po klatce. Taka analiza za pomocą ręcznego kodowania twarzy zajęłaby ponad 60 godziny analizy (przy założeniu, że człowiek kodowałby 1 klatkę w 5 sekund). FaceReader zrobił taką szczegółową analizę w zaledwie 30 minut, na rysunkach i analizach poniżej przedstawiane są wyniki.

Emocje Prezydenta Andrzej Duda

Expression_AllExpression_Neutral_Removed

Rysunek 2. Dystrybucja emocji Prezydenta Andrzej Duda.

Patrząc na górny obrazek na Rysunku 2, można stwierdzić, iż Andrzej Duda nie używał swojej mimiki twarzy często. Podczas swojego wystąpienia miał przeważnie neutralną ekspresję twarzy (87%).

Przeprowadzona została również bardziej szczegółowa analiza, w której nie brano pod uwagę stan neutraly. Na dolnym obrazku na Rysunku 2 można zauważyć, że istnieje duża różnica pomiędzy emocjami. Najczęstsza emocja Andrzeja Dudy to była radość (57%) i zaskoczenie (34%).

Co więcej, w Wideo 1 (patrz poniżej), możemy zobaczyć część analizy FaceReader, gdzie Prezydent wyraźnie pokazuje radość w odpowiedzi do pytania jednego z internautów „Jak się Pan czuje jak Prezydent Polski?’’(14 sekunda).

Ponadto w Wideo 2, możemy zobaczyć, jak w ciągu 90 sekund przemówienia, od 00:46.39- 02:09.25, szybkość bicia serca spada z 89 uderzeń na minutę do 74 uderzeń na minutę.

Video 1 – Niewielka część z wideo z przejawami radości

Video 2 – Niewielka część z wideo z analiza bicia serca

Co ważne Prezydent Andrzej Duda powiedział pod koniec swojego wystąpienia, że będzie korzystać z Facebooka także w przyszłości. Będzie więc można przeanalizować jego nastepnę przemówienia w ten sam sposób.

Podsumowanie:

– Prezydent miał przeważnie neutralną ekspresję twarzy (87% czasu)
– Jak już emocję pokazywał to dwa razy wiecej radość (57%) niż zaskoczenia (34%), co może prowadzić do wniosku iż generalnie podobały mu się pytania internatów, ale niektóre go zapewne zaskoczyły
– Pan Prezydent wyraźnie pokazał emocję radości w odpowiedzi do pytania jednego z internautów „Jak się Pan czuje jak Prezydent Polski?’’.
– Szybkość bicia serca spadła najszybciej (z 89 uderzeń na minutę do 74 uderzeń na minutę) na poczatku przemówienia od (00:46- 02:09), co może prowadzić do wniosku iż na początku trochę denerwował.

Jak działa FaceReader

FaceReader to narzędzie które automatycznie analizuje mimikę twarzy, zapewniając użytkownikom obiektywną ocenę emocji twarzy danej osoby [1], [2], [3], [4]. Twarz osoby jest wykorzystywana do analizy w ramach uczenia maszynowego, gdzie ponad 500 punktów na twarzy używane jest do uzyskania modelu 3D twarzy. Taki Model 3D jest dalej wykorzystywany w celu uzyskania emocji. Ponad 500 punktow jest śledzonych i analizowanych w czasie rzeczywistym w celu uzyskania emocji na twarzy podczas całego przemówienia. Poniższy film przedstawia jak działa ten system. Aby uzyskać więcej informacji prosimy odwiedzić stronę internetową Noldus FaceReader.

FaceReader został naukowo przetestowany i posiada następującą dokładność w rozpoznawaniu podstawowych emocji: 96 % – Radość, 86% – Smutek, 76% – Zlość, 94% – Zaskoczenie, 82 % – Strach, 92 % – Wstręt. Więcej informacji znajduje się w publikacji naukowej na ten temat w międzynarodowym czasopiśmie naukowym w Lewinski i wsp. (2014).

FaceReader w wyborach w Ameryce i w Turcji

FaceReader zastosowano również w analizie politycznych debat, np. w ostatnich wyborach w USA, zobacz film na CNN i w ostatnich wyborach w Turcji.

 

Bibliografia:

Ekman, P., Friesen W.V., & Hager J. C. (2002). Facial action coding system: The manual. Salt Lake City, UT: Research Nexus.

Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969). Pan-cultural elements in facial displays of emotion. Science, 164(3875), 86-88. doi:10.1126/science.164.3875.86

Russell, J. A. (1980). A circumplex model of affect. Journal of personality and social psychology, 39(6), 1161.

FaceReader. (2015). FaceReader: Tool for automatic analysis of facial expression: Version 6.1. Wageningen, the Netherlands: Noldus Information Technology B.V.

Lewinski, P., den Uyl, T. M., & Butler, C. (2014). Automated facial coding: Validation of basic emotions and FACS AUs in FaceReader. Journal of Neuroscience, Psychology, and Economics, 7(4), 227.

[1] Kuilenburg, H. van, Wiering, M. and Uyl, M.J. den (2005). A Model Based Method for Automatic Facial Expression Recognition. In Proceedings of the 16th European Conference on Machine Learning (ECML-2005), October 3-7, Porto, Portugal.

[2] Marten J. den Uyl, Hans van Kuilenburg, (2005). The FaceReader: Online facial expression recognition. Proceedings of Measuring Behaviour 2005, 5th International Conference on Methods and Techniques in Behavioural Research, 589-590.

[3] H. Emrah Tasli, Amogh Gudi, and Marten den Uyl, “Remote PPG based vital sign measurement using adaptive facial regions,” in International Conference on Image Processing, 2014.

[4] H. Emrah Tasli, Paul Ivan; “Turkish Presidential Elections TRT Publicity Speech Facial Expression Analysis”; arXiv:1408.3573

Zastrzeżenie:

Wszystkie filmy źródłowe i dane są publicznie dostępne do celów badawczych. W razie potrzeby, analiza może zostać powtórzona. Wszystkie wyniki nie stanowią odzwierciedlenia jakichkolwiek osobistych poglądów politycznych. Celem niniejszej pracy jest analiza ekspresji twarzy Prezydenta podczas jego przemówień.

Ważne jest tez to ze analiza przedstawiona w tym badaniu pokazuje jaka emocje są na twarzy. Analiza nie ma zamiaru stwierdzić, jak osoba czuła się w trakcie przemówienia. Osoba mogła czuć się inaczej niż analiza to pokazuje. Analiza jest zaprezentowana jako praca naukowa a co za tym idzie, nie wyraża osobistych przekonań.

Oryginalne i publicznie dostępny wideo można znaleźć tutaj.

 

 

Advertisements
Standard

Facial expression analysis of President dr. Andrzej Duda’s talk on “Facebook Mentions” on 06-Aug-2015

Wersja po polsku tutaj.

Newly appointed Polish President’s speech on Facebook Mentions addressed to Polish internet users was analyzed with automated facial coding. The speech was streamed online on 6th of August 2015, just to 2.5 hours after the official appointment.

In this facial expression analysis, the three newest developments were combined: a) first official speech of a new Polish president who was b) talking on a new Facebook’s service and c) applying a new method – automated facial coding software, called FaceReader (2015).

The video lasted 25 minutes, 10 seconds, which allowed to analyze more than 45’000 frames, frame by frame. Such analysis by manual facial coding would take more than 60 hours to analyze (assuming that a human would code 1 frame in 5 seconds). FaceReader Software was able to provide such detailed analysis in just 30 minutes.

FaceReader Analysis: FACS, Heart Rate, Valence/Arousal & Basic Emotions

FR_view

Figure 1. Visualization of the analysis in FaceReader.

Figure above shows that FaceReader provides four different measures from the face. The numbers on the face – 2, 6, 7 and 12 with corresponding intensity ratings (the darker the color, the more intense the expression) – are Action Units (AUs) based on Facial Action Coding System (Ekman et al., 2002). The red heart with a number (in this case 84) indicated beats per minutes (bpm), which is a measure of heart rate. Next, the Circumplex Model of Affect (Russel, 1980) demonstrates valence (pleasant/unpleasant emotion) and arousal (active/inactive emotion). The expression line at the bottom indicate Basic Emotions (Ekman et al. 1969) (Happy, Sad, Angry, Surprise, Scared, Disgust)

Emotions of President Andrzej Duda

Expression_AllExpression_Neutral_Removed

Figure 2. Distribution of emotions of President Andrzej Duda.

Looking at the upper-side of the figure, one can conclude that Andrzej Duda do not use his facial expressions much and he generally has a neutral facial (87%) state during his speech.

A more detailed analysis was also performed where the neutral state is discarded. We can see at the bottom-side of the figure that there is a major difference between the emotions showed. It is observed that the most common expression of Andrzej Duda is happiness with 57% and surprise is following that with 34%. In Video 1, we can see a part of the analysis where those emotions are show. Further in Video 2, we can see how over 90 seconds, from 00:46.39 to 02:09.25 of his speech, his heart beat rate goes down from 89 beats per minute to 74 beats per minute.

Video 1 – a small part from the video with expressions of happiness

Video 2 – a small part from the video with heart beat analysis.

How FaceReader works

FaceReader is a tool to automatically analyzing facial expressions, providing users with an objective assessment of a person’s facial emotions [1], [2], [3], [4]. The image of the person is used for the analysis in a machine learning framework where a training based facial landmark tracking is performed for obtaining a 3D model of the face. The 3D model is further used to obtain the facial expressions in a supervised learning framework. The video below demonstrates how the system works. For more information please visit Noldus FaceReader website. Over 500 landmark points are tracked and analyzed for obtaining the facial expressions during the whole speech.

The FaceReader tool has been validated, showing that the accuracy for emotion recognition is as follows: 96 % – Happiness, 86% – Sadness, 76% – Anger, 94% – Surprise, 82 % – Fear, 92 % – Disgust (Lewinski et al, 2014)

FaceReader in U.S. and Turkish elections

FaceReader was also applied in analyzing political debates, in the last U.S. election, please see a video on CNN and with the latest Turkish presidential elections.

References:

Ekman, P., Friesen W.V., & Hager J. C. (2002). Facial action coding system: The manual. Salt Lake City, UT: Research Nexus.

Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969). Pan-cultural elements in facial displays of emotion. Science, 164(3875), 86-88. doi:10.1126/science.164.3875.86

Russell, J. A. (1980). A circumplex model of affect. Journal of personality and social psychology, 39(6), 1161.

FaceReader. (2015). FaceReader: Tool for automatic analysis of facial expression: Version 6.1. Wageningen, the Netherlands: Noldus Information Technology B.V.

Lewinski, P., den Uyl, T. M., & Butler, C. (2014). Automated facial coding: Validation of basic emotions and FACS AUs in FaceReader. Journal of Neuroscience, Psychology, and Economics, 7(4), 227.

[1] Kuilenburg, H. van, Wiering, M. and Uyl, M.J. den (2005). A Model Based Method for Automatic Facial Expression Recognition. In Proceedings of the 16th European Conference on Machine Learning (ECML-2005), October 3-7, Porto, Portugal.

[2] Marten J. den Uyl, Hans van Kuilenburg, (2005). The FaceReader: Online facial expression recognition. Proceedings of Measuring Behaviour 2005, 5th International Conference on Methods and Techniques in Behavioural Research, 589-590.

[3] H. Emrah Tasli, Amogh Gudi, and Marten den Uyl, “Remote PPG based vital sign measurement using adaptive facial regions,” in International Conference on Image Processing, 2014.

[4] H. Emrah Tasli, Paul Ivan; “Turkish Presidential Elections TRT Publicity Speech Facial Expression Analysis”; arXiv:1408.3573

Disclaimer:

All source videos and the data are publicly available for research purposes. If required, the analysis can be repeated and the results can be validated. All these results are obtained scientifically and they do not represent or reflect any personal view or political opinion. The purpose of this study is to analyses the facial expressions of the president during his speeches.

It is important to further emphasize that the analysis presented in this study shows how the president looks like. There is no intention to claim how the president would feel during the speech. This work is presented from a scientific point of view and hence, no personal option is reflected.

The original and publicly available video can be found here.

Standard

Turkish presidential elections TRT publicity speech facial expression analysis (03.08.2014)

Türkçesi için buraya tıklayınız.

In this study, facial expressions of the three Turkish presidential candidates Demirtas, Erdogan and Ihsanoglu (in alphabetical order) are analyzed during the publicity speeches featured at TRT (Turkish Radio and Television) on 03.08.2014. FaceReader is used for the analysis where 3D modeling of the face is achieved using the active appearance models. Over 500 landmark points are tracked and analyzed for obtaining the facial expressions during the whole speech. All source videos and the data are publicly available for research purposes. If required, the analysis can be repeated and the the results can be validated. All these results are obtained scientifically and they do not represent or reflect any personal view or political opinion. The purpose of this study is to analyse the facial expressions of the presidential candidates during their campaign speeches.

How the System Works

FaceReader is the world’s first tool capable of automatically analyzing facial expressions, providing users with an objective assessment of a person’s facial emotions [1], [2], [3], [4]. The image of the person is used for the analysis in a machine learning framework where a training based facial landmark tracking is performed for obtaining a 3D model of the face. The 3D model is further used to obtain the facial expressions in a supervised learning framework. The video below demonstrates how the system works. For more information please visit Noldus FaceReader website.

Sample analysis of the three candidates (Demirtaş, Erdoğan, İhsanoğlu).

 

candidates

Selahattin Demirtaş

A small part from the video and the analysis result:

Demirtas3E

 

Recep Tayyip Erdoğan

A small part from the video and the analysis result:

Erdogan3E

 

Ekmeleddin İhsanoğlu

A small part from the video and the analysis result:

Ihsanoglu3E

 

 What does the Figures say?

On the left column, the average facial expression of each of the candidates for the 10-15 min video is presented. On the right hand, similar average facial expressions are presented when the neutral moments (gray region on the left) is discarded.

CONCLUSION

First of all, it is important to emphasize that the analysis presented in this study shows how the candidates look like. There is no intention to claim how the candidates would feel during the speech. This work is presented from a scientific point of view and hence, no personal option is reflected.

The three candidate public speech videos of length 10-15 min are analyzed. We would like to share some remarkable observations:

Looking at the left column of the figure, one can conclude that the candidates do not use their facial expressions a lot and they generally have a neutral facial state during their speeches. As a comparison, Ihsanoglu stands outs as the candidate who uses his mimics the most with 32%, Erdogan follows with 15% and Demirtas is the one with the least usage of his mimics with 6%.

Detailed analysis of the part where the neutral state is discarded we can see in the right column that there is a major difference between the candidates. It is observed that the common expression of Erdogan is anger with 82% and fear is following that with 10%. Similarly, the analysis of Ihsanoglu shows that, anger is the most common expression with 74% and disgust follows that with 20%. According to the analysis of Demirtas, the most common expression is happiness with 50% and fear is following that with 34%.

Such an analysis would definitely be more interesting during a real time discussion on a TV program where all the candidates are present.  However, under the current conditions, this is the best possible unbiased way to have a comparative analysis during the publicity speech where TRT is officially liable to broadcast. We hope to make a future study during a real time broadcast with all the candidates present. You can find the contact form below.

 

[1] Kuilenburg, H. van, Wiering, M. and Uyl, M.J. den (2005). A Model Based Method for Automatic Facial Expression Recognition. In Proceedings of the 16th European Conference on Machine Learning (ECML-2005), October 3-7, Porto, Portugal.
[2] Marten J. den Uyl, Hans van Kuilenburg, (2005). The FaceReader: Online facial expression recognition. Proceedings of Measuring Behaviour 2005, 5th International Conference on Methods and Techniques in Behavioural Research, 589-590.
[3] H. Emrah Tasli, Amogh Gudi, and Marten den Uyl, “Remote PPG based vital sign measurement using adaptive facial regions,” in International Conference on Image Processing, 2014.
[4] H. Emrah Tasli, Paul Ivan; “Turkish Presidential Elections TRT Publicity Speech Facial Expression Analysis”; arXiv:1408.3573
Standard

Cumhurbaşkanlığı seçimi TRT propaganda konuşması yüz mimik analizi (03.08.2014)

For English version click here.

Bu çalışmada 03.08.2014 tarihli TRT de yayimlanan Cumhurbaskanlığı seçim konuşmalarının yüz analizi yapılmıştır. Analiz için FaceReader yazılımı kullanılmış olup video linkleri ve yazılımın sayfası paylaşılmıştır. İstenildiği takdirde testler tekrarlanarak sonuçların doğruluğu kontrol edilebilir. Buradaki tüm sonuçlar tamamen bilimsellik kriterlerine bağlı kalarak oluşturulmuş olup herhangi bir kişisel veya siyasi görüşü yansıtmamaktadır. Amaç objektif kriterler altında Cumhurbaşkanı adaylarının propaganda konuşmalarında yansıttıkları yüz mimiklerinin analizini yapmaktır. Sayfa içi sıralama adayların soy isimlerine göre alfabetik olarak yapılmıştır.

Sistem nasıl çalışıyor?

FaceReader otomatik yüz analizi için geliştirilmiş ilk ürün olarak yüz ifadelerinin objektif olarak değerlendirilmesine olanak sağlamaktadır [1], [2], [3], [4]. Kişinin görüntüsü makina öğrenimi temelli bir yapı altında analiz edilerek yüzdeki önemli noktaların bulunması ile yüze ait 3 boyutlu bir model oluşturmasını temel almaktadır. Aşağıdaki video sistemin çalışmasını göstermektedir. Daha fazla bilgi için Noldus sitesi ziyaret edilebilir.

 

Üç adayın yüz analizinden bir kare:

candidates

 

Aşağıda analizi yapılan 10-15 dakikalık videolardan 30 saniyelik örnekler bulunmaktadır.

Selahattin Demirtaş

İlgili video dan bir kesit ve analizi:

Demirtas3

Recep Tayyip Erdoğan

İlgili video dan bir kesit ve analizi:

Erdogan3

Ekmeleddin İhsanoğlu

İlgili video dan bir kesit ve analizi:

Ihsanoglu3

Grafikler ne söylüyor?

Sol kolondaki grafikte her adaya ait 10-15 dakikalık konuşmadan elde edilen ortalama yüz ifadesi görülmektedir. Sağ kolondaki grafikte ise aynı zaman kesiti için adayların nötr oldukları zamanlar çıkarıldığında (soldaki gri bölge) elde edilen ortalama yüz ifadeleri gösterilmektedir.

SONUÇLAR

Öncelikle şu noktayı vurgulamakta fayda var. Buradaki analiz adayların yüz mimiklerinden nasıl göründüklerine yönelik bir çalışmanın ürünüdür. Adayların konuşma sırasında nasıl hissettiğine yönelik bir iddiada bulunulmamaktadır. Tüm çalışma bilimsel kriterler gözetilerek yapılmıştır ve kişisel hiçbir görüşü yansıtmamaktadır.

Sonuçlar incelendiğinde dikkat çeken birkaç önemli gözlemi paylaşmak gerekirse:

* Soldaki grafiğe bakılınca genel olarak adayların mimiklerini çok fazla kullanmadığı ve konuşmanın genelinde nötr bir yüz ifadesine sahip olduğu söylenebilir. Karşılaştırmak açısından, İhsanoglu %32 ile en çok mimik kullanan aday olarak göze çarpmaktadır. Takiben %15 ile Erdoğan ikinci sırada yer alırken, Demirtaş %6 ile en az mimik kullanmaktadır.

* Sağ kolondaki grafiğe göre, adayların nötr ifadeleri çıkartıldığında elde edilen sonuçlar ise şu şekilde: Demirtaş’ın en yoğun ifadesinin %50 ile mutluluk ve onu takiben %34 ile korku olduğu gözlemleniyor. Erdoğan’ın  en yoğun ifadesi %82 ile kızgınlık ve onu takiben %12  ile korku olduğu görülüyor. Son olarak İhsanoğlu’nun analizinde %74 kızgınlık en yoğun ifade olarak gözükürken %20 ile memnuniyetsizlik takip ediyor.

Bu tip bir analiz, adayların aynı ortamda bulunduğu bir tartışma programında daha bilgi verici olabilirdi, fakat günümüz koşullarında yapılabilecek en eşit karşılaştırmayı TRT nin yasal olarak vermekle yükümlü olduğu propaganda konuşmalarından edinmek mümkün oldu. İlerde böyle bir çalışmayı canlı yayın sırasında tüm adayların bulunduğu bir ortamda gerçekleştirmek dileğiyle. İletişim için:

 

[1] Kuilenburg, H. van, Wiering, M. and Uyl, M.J. den (2005). A Model Based Method for Automatic Facial Expression Recognition. In Proceedings of the 16th European Conference on Machine Learning (ECML-2005), October 3-7, Porto, Portugal.
[2] Marten J. den Uyl, Hans van Kuilenburg, (2005). The FaceReader: Online facial expression recognition. Proceedings of Measuring Behaviour 2005, 5th International Conference on Methods and Techniques in Behavioural Research, 589-590.
[3] H. Emrah Tasli, Amogh Gudi, and Marten den Uyl, “Remote PPG based vital sign measurement using adaptive facial regions,” in International Conference on Image Processing, 2014.
[4] H. Emrah Tasli, Paul Ivan; “Turkish Presidential Elections TRT Publicity Speech Facial Expression Analysis”; arXiv:1408.3573

 

Standard