142,99 €
Mehr erfahren.
- Herausgeber: John Wiley & Sons
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
Automatically evaluating the aesthetic qualities of a photograph is a current challenge for artificial intelligence technologies, yet it is also an opportunity to open up new economic and social possibilities. Aesthetics in Digital Photography presents theories developed over the last 25 centuries by philosophers and art critics, who have sometimes been governed by the objectivity of perception, and other times, of course, by the subjectivity of human judgement. It explores the advances that have been made in neuro-aesthetics and their current limitations. In the field of photography, this book puts aesthetic hypotheses up against experimental verification, and then critically examines attempts to "scientifically" measure this beauty. Special attention is paid to artificial intelligence techniques, taking advantage of machine learning methods and large databases.
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 555
Veröffentlichungsjahr: 2023
Ähnliche
Table of Contents
Cover
Title Page
Copyright Page
Introduction Image and Gaze
Mechanisms related to interest
Mechanisms associated with surprise
Pleasure
versus
arousal
Art, Beauty and Aesthetics – how are they related?
Our work
1 The Legacy of Philosophers
1.1. The objectivist approach
1.2. The subjectivist approach
1.3. Subjectivism and objectivism: an ongoing debate
2 Neurobiology or the Arbitrator of Consciousness
2.1. fMRI protocols and neuroaesthetics
2.2. The fMRI quest for “beauty processes” in the brain
2.3. Responses from functional electric encephalography
2.4. A global cognitive scheme for aesthetic judgment?
2.5. A critique of neuroaesthetic methods
3 What Are the Criteria For a Beautiful Photo?
3.1. Before we enter into the fray
3.2. Composition
3.3. Histograms, spectral properties and textures
3.4. Color
3.5. What behavioral psychosociology has to say
4 Algorithmic Approaches to “Calculate” Beauty
4.1. First steps: C. Henry
4.2. G.D. Birkhoff’s mathematical approach
4.3. Those who followed G.D. Birkhoff
4.4. Algorithmic approach with AI: J. Schmidhuber
5 The Holy Grail of the Digital World: Artificial Intelligence
5.1. Which artificial intelligence?
5.2. Why artificial intelligence in aesthetics?
5.3. Expert opinions
5.4. The database
6 Primitive-based Classification Methods
6.1. Judging aesthetics
6.2. Help in composing beautiful photos
6.3. Some specific research related to the evaluation of aesthetics using primitives
7 Deep Neural Network Systems
7.1. DNNs dedicated to aesthetic evaluation
7.2. Variants around the basic DNN architecture
7.3. Written appraisals: analyzing them and formulating new ones
7.4. Measuring subjective beauty
8 A Critical Analysis of Machine Learning Techniques
8.1. The popularity of studies on aesthetics
8.2. A summary of learning methods
8.3. Questioning the hypotheses
8.4. Specific features of beautiful images detected by a computer
Conclusion
Appendix 1: A Brief Review of AestheticsA Brief Review of Aesthetics
A1.1. Aesthetics in the ancient world: objectivism ruled supreme
A1.2. The Renaissance
A1.3. The modern world: from objectivism to subjectivism
Appendix 2: Aesthetics in ChinaAesthetics in China
A2.1. The image in Chinese literature
A2.2. Objective or subjective?
A2.3. Artificial intelligence and the aesthetic appraisal of Chinese art
Appendix 3: The Aesthetic of Persian MiniaturesThe Aesthetic of Persian Miniatures
A3.1. A brief history
A3.2. What is the aesthetic of the miniature?
A3.3. Objective or subjective?
Appendix 4: Aesthetics in JapanAesthetics in Japan
A4.1. A brief history of art in Japan
A4.2. The art of impermanence
References
Index
End User License Agreement
List of Tables
Chapter 5
Table 5.1
Proposal for a hierarchy of photographs
Table 5.2
Correspondence between Fang’s classification (Fang and Zhang 2017)
...
Table 5.3
Some databases that are often used to study the aesthetics of imag
...
Table 5.4
The 14 style attributes used in the AVA database and the number of
...
Table 5.5
Effect that various transformations have on the appraisal score of
...
Chapter 6
Table 6.1
An example of the primitives chosen for a study of aesthetics in p
...
Chapter 7
Table 7.1
Results of the learning through DNN
Table 7.2
Comparison of the performances of evaluation of images in the AVA
...
Table 7.3
The role of semantic information in the performances of the networ
...
Chapter 8
Table 8.1
The various machine learning systems of phototgraphic quality in t
...
Table 8.2
Comparative results of different methods of aesthetic evaluation i
...
List of Illustrations
Introduction
Figure I.1
An image’s relevance may be related to the general context of kno
...
Figure I.2
Amazement is another mechanism that leads us to pay particular at
...
Figure I.3
Both these photographs are indisputably remarkable for their aest
...
Figure I.4
The affective space defined by Lang et al.
Figure I.5
Art that is not very aesthetic.
Chapter 1
Figure 1.1
Are these photos beautiful. This is the question this text attemp
...
Figure 1.2
Why is a photo beautiful?
Chapter 2
Figure 2.1
(a) Diagram of the optic pathways that go from the eyes to the vi
...
Figure 2.2
Active zones in Ishizu and Zeki’s experiment.
Figure 2.3
Zones active when emotions are manifested in the experiment by Ch
...
Figure 2.4
The chief focal areas of visual activity during the experiments r
...
Figure 2.5
Experiment carried out by Cupchik et al. (2009).
Figure 2.6
In this experiment from Cupchik et al. (2009), we can see the res
...
Figure 2.7
Two results from EEG analysis showing the role of different brain
...
Figure 2.8
A. Chatterjee’s functional model for vision with aesthetic judgme
...
Figure 2.9
Functional model of connectivity by Brown et al. for vision with
...
Figure 2.10
A model of aesthetic perception by H. Leder (from Leder et al. (
...
Figure 2.11
Functional model by C. Redies.
Figure 2.12
The functional model (called the quartet model) by S. Koelsch et
...
Figure 2.13
Double-intentional object model by L.H. Hsu
Chapter 3
Figure 3.1
“Quality” and “beauty” of a photograph.
...
Figure 3.2
The figures used in VAST for aesthetic measurements
Figure 3.3
Elements of the aesthetic judgments of landscapes from (Hunter an
...
Figure 3.4
Several effects listed in this text come together in this landsca
...
Figure 3.5
(a) Oculometry carried out on a painting by Vladimir Repin (
Unexp...
Figure 3.6
Grids that make it possible to detect the composition of the imag
...
Figure 3.7
The rule of thirds shows how to arrange the zones of attention wi
...
Figure 3.8
Aesthetic quality depending on adherence to the rule of thirds (f
...
Figure 3.9
(a) Tight framing is required for an object with very regular mot
...
Figure 3.10
Using a small range of gray levels is not generally recommended,
...
Figure 3.11
The role played by the histogram’s bias (third order moment) on
...
Figure 3.12
In many images, a precise focus on all objects in the image is a
...
Figure 3.13
The texture perception model by Thumfart et al. (2011).
Figure 3.14
Color wheels proposed by different authors.
Figure 3.15
Harmony of colors.
Figure 3.16
The work carried out by Moon and Spencer: (a) Munsell’s space of
...
Figure 3.17
The various types of chromatic harmony according to Moon and Spe
...
Figure 3.18
The configurations chosen by Matsuda. The chromatic circle can r
...
Figure 3.19
Beauty in magazines
Chapter 4
Figure 4.1
Application of Birkhoff’s formula to a vase.
Figure 4.2
In their research, Rigau et al. chose an experimental base of hig
...
Figure 4.3
Image (a) is beautiful because it is simple. J. Schmidhuber sugge
...
Chapter 5
Figure 5.1
Relationship between the scores for aesthetics and originality in
...
Figure 5.2
Scattering of the average scores awarded to each photo in the DPC
...
Chapter 6
Figure 6.1
(a) Distribution of scores awarded by the ACQUINE software to a c
...
Figure 6.2
Paintings that received a score of 10/10 on the ACQUINE evaluatio
...
Figure 6.3
Other paintings from the classical repertoire which received a sc
...
Figure 6.4
Other paintings from the classical repertoire that received a sco
...
Figure 6.5
How to separate beautiful and less beautiful photos?
Figure 6.6
Comparison of the results (accuracy based on recall) from Ke and
...
Figure 6.7
Detection of primitives of motives (on top) and composition (at t
...
Figure 6.8
Schema for computing the aesthetic, developed by Dhar et al.
Figure 6.9
Comparison of the results obtained used Ke’s approach and Dhar’s
...
Figure 6.10
Performances using the approach proposed by Marchesotti et al.
Figure 6.11
The analysis windows proposed by Su et al., which supply the lib
...
Figure 6.12
The approach used by Lo et al. (a) The final judgment is derived
...
Figure 6.13
Rate of correct recognition during the learning of color harmony
...
Chapter 7
Figure 7.1
A rectangular image (left) can be adapted for the square input la
...
Figure 7.2
The two-path neuromimetic network (RAPID) proposed by Lu et al.
Figure 7.3
Scheme for the learning in the deep, multi-path neural network
Figure 7.4
Merging of the multi-path network: the orderless multi-patch aggr
...
Figure 7.5
The two architectures proposed by Mai et al. (a) To address the p
...
Figure 7.6
The various components in the architecture used in Kong et al. (2
...
Figure 7.7
Architecture of the A-Lamp system. This highlights, in the upper
...
Figure 7.8
The A-Lamp system. (a) The multi-column sub-network processes, in
...
Figure 7.9
How did the Internet vote?
Figure 7.10
The architecture used in the first step of the research by Kao e
...
Figure 7.11
Architecture of the BDN system developed by Wang et al., inspire
...
Figure 7.12
The Creatism system, beginning with the 360º panorama (b), looks
...
Figure 7.13
Construction of comments from the PCCD database (Chang et al. 20
...
Figure 7.14
Results of the aesthetic evaluation of four images in the AVA da
...
Figure 7.15
The five criteria for judgment in the AMAN system (on the left)
...
Figure 7.16
Scheme of the AAMAN system by Jin et al
Figure 7.17
Architecture of the system developed by Li et al. It uses two br
...
Figure 7.18
Two images projected in the space of the Big Five. The five scor
...
Figure 7.19
Scheme of the USAR system by Lv et al.
Figure 7.20
Detection of communities in the cloud of descriptors using a lat
...
Chapter 8
Figure 8.1
Performances of two-class aesthetic evaluation (beautiful/not bea
...
Figure 8.2
A few examples of “abstract” photographs (or weakly figurative),
...
Figure 8.3
The origins of photos in the AVA database. Two-thirds of the phot
...
Figure 8.4
Twelve photos randomly drawn from the AVA library from among thos
...
Figure 8.5
The 16 “most beautiful” images in the AVA library, arranged in de
...
Figure 8.6
Five photos among the lowest scored images in the AVA database (t
...
Appendix 1
Figure A1.1
There are no longer paintings available that date back to this p
...
Figure A1.2
Evolution of representations from the Renaissance to Classicism.
Figure A1.3
The serpentine line appears on the original cover of Hogarth’s
...
Appendix 2
Figure A2.1
Making visible the invisible, expressing the fluidity of water,
...
Figure A2.2
Drawing with emptiness, using simple and efficient lines to evok
...
Figure A2.3
Simplifying lines and letting the viewer guess at the absent; th
...
Figure A2.4
The Five Phases (or wuxing) theory associated with the fundament
...
Appendix 3
Figure A3.1
Islam’s rules about depicting living objects led to fantastic ab
...
Figure A3.2
Some miniatures from the Persian School
Figure A3.3
Some miniatures from the Persian school
Appendix 4
Figure A4.1
Two examples of nise-e. (a) Fujiwara Nobuzane (1176–1266),
...
Figure A4.2
Sengai Gibon (1750–1837) is famous for this piece,
Circle...
Figure A4.3
Hakuin Ekaku, Zen Buddhist philosopher and painter (1686–1769),
...
Figure A4.4
(a) Ito Jakuchu (1716–1800),
Mandarin ducks in the snow
.
...
Figure A4.5
(a) Ogata Kōrin (called Ichinojō) (1658–1716),
...
Figure A4.6
(a) and (b) Tosa Mitsuoki (1617–1691),
Scenes of Genji Mo...
Guide
Cover Page
Title Page
Copyright Page
Introduction
Table of Contents
Begin Reading
Conclusion
Appendix 1 A Brief Review of Aesthetics
Appendix 2 Aesthetics in China
Appendix 3 The Aesthetic of Persian Miniatures
Appendix 4 Aesthetics in Japan
References
Index
Wiley End User License Agreement
Pages
iii
iv
xi
xii
xiii
xiv
xv
xvi
xvii
xviii
xix
xx
xxi
xxii
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
237
238
239
240
241
242
243
244
245
246
247
248
249
151
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
789
296
297
298
299
300
Series EditorMarie-Christine Maurel
Aesthetics in Digital Photography
Henri Maître
First published 2023 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentioned address:
ISTE Ltd
John Wiley & Sons, Inc.
27-37 St George’s Road
111 River Street
London SW19 4EU
Hoboken, NJ 07030
UK
USA
www.iste.co.uk
www.wiley.com
© ISTE Ltd 2023The rights of Henri Maître to be identified as the authors of this work have been asserted by him in accordance with the Copyright, Designs and Patents Act 1988.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s), contributor(s) or editor(s) and do not necessarily reflect the views of ISTE Group.
Library of Congress Control Number: 2022946611
British Library Cataloguing-in-Publication DataA CIP record for this book is available from the British LibraryISBN 978-1-78630-753-8
IntroductionImage and Gaze
The aesthetic emotion, which gives rise to the impression of beauty, is undoubtedly universal in humans.
Edgar MORIN (2013, p. 13)
The proliferation of images in the modern world increasingly forces us to select photos from among a vast set. The task of choosing images to illustrate an article, a book cover, a poster, or one that represented an event or a voyage used to be entrusted only to professionals in the fields of publishing, communication, archives, or professional photographers and collectors. Today, it is incumbent on each of us to decide what we wish to preserve and what to forget, what we wish to send, post on the Internet, delete forever, or to archive (in the improbable event of us ever revisiting it). Managing photo archives is a real bane for many us1. The step of selecting images has always been assumed to be delicate and of great importance. In the professional context, it is entrusted to those who are renowned as experts or in a position of authority.
Let us pause for a moment to examine the mechanisms that are activated when we make decisions about sorting and selecting images. Where does the beauty of the image fall in our hierarchy?
It would seem that the primary mechanisms that are activated when selecting photographs can be divided into three broad categories:
– why the document is of interest, that is, its ability to draw and hold our attention by relating the document to contexts familiar to us;
– the surprise factor, that is, contrary to the previous point, its ability to give us a novel visual or cognitive experience by bringing in an unexpected contribution;
– beauty, that is, the pleasure it brings us, independent of its content, through the arrangement of its elements.
The last point is what we will be looking at, exclusively, in this book.
In this framework, the first two points are the result of associative impressions, as used by G.T. Fechner, who used this distinction from the 19th century (Fechner 1871). It often happens that the same image associates several of these registers, with its attractiveness being heightened, but with the contribution of each register of attention being less clear. However, with respect to our decision-making, the contributions of interest, surprise and beauty would seem to evolve2, in independent or orthogonal spaces, as used by Gärdenfors (2000), that is, without any intimate influence on each other. They are therefore evaluated separately by our consciousness, and then probably combined into a single score that ultimately makes us prefer one image to another in a heuristic choice that is difficult to express, but which is likely to follow the empirical decision-making schemas proposed by Tversky and Kahneman (1981).
Before we move away from them, let us briefly study the first two domains. In the field of image processing, these are sometimes referred to by other names as well: interestingness3, memorability4, unusualness5, or popularity6; we will illustrate these with examples (Isola et al. 2011; Gygli et al. 2013; Amengual et al. 2015).
Mechanisms related to interest
These mechanisms have been of particular interest in studies among experimental psychologists from the 1920s onwards. The work produced by these psychologists is often brought together under the umbrella term relevance theory (Sperber and Wilson 2004).
Interest or relevance, that is, a complex psycho-physiological experience, may be produced by:
– external stimuli (environmental stimuli transmitted by our senses) or internal stimuli (biochemical, created by our cortex). It is therefore a passive process;
– or through reasoning, that is, cognitive processes that produce new elements of knowledge from a particular context. This is then an active process.
W. James’ work (Lange and James 1922) pioneered studies on the relation between emotion and how a person appraises a situation. The importance of arousal and the order of the various steps involved (arousal, emotion, appraisal) has also been well documented. This research has served as a guide for various studies conducted 50 years later in the field of aesthetics.
However, it was Sperber and Wilson (2004), above all, who made it possible to construct a relevance theory, bringing together all the cognitive baggage of the recipient and not only their linguistic knowledge, as suggested by W. James. In this context, Dessalles (2008) suggested a more quantitative measure of relevance and endowed it with predictive capacities using an original mathematical model.
If we want to apply this relevance theory to the interest aroused by a collection of images, it is useful to define two extreme cases that may be approached in different ways:
– universal interest refers to themes that are often displayed in society and media and transmitted through context and culture: such-and-such an actor or sportsperson, a car, monument or an event that is explicitly and regularly covered. The popularity of these themes can today be measured using mediametry tools (Hsieh et al. 2014; Fu et al. 2014): frequency of exposure on television, the number of instances on the Internet, “clicks” or “likes” on social media, etc. (Figure I.1(a));
– personal interest brings one closer to the images because the themes are deeply connected to individual life, more specifically, the viewer’s personal life: “my” family, “my” city, “my” work, etc. The evaluation of relevance then takes the conventional forms proposed in Dessalles (2013), for example, which involves variables of space and time that decrease more or less rapidly, and relations in the degree of proximity (e.g. in a family tree or a company flow chart) (Figure I.1(b)).
Figure I.1An image’s relevance may be related to the general context of knowledge within a community ((a) actors, sportspersons, objects that frequently appear in newspapers or social media), or, on the contrary, to the personal context of the observer ((b) members of the same family, vacation sites, hobbies, etc.).
Mechanisms associated with surprise
These mechanisms function a little differently from those mentioned above, as they become more effective when the document presented to us is more removed from the familiar (Figure I.2). Surprise or amazement (as this is a broader term, it may be more useful for us here as it covers a wider range of responses) can be narrowed down to various forms: humor, fear, perplexity, affection, disgust, etc.
It can be clearly seen that relevance, on the one hand, and surprise, on the other, are different from aesthetic qualities that motivate us to remember a photograph. It is also quite likely that they operate together to lead to a choice7, as can be seen in the two photos in Figure I.3. However, in the following sections, we will do our best to separate these two mechanisms. That is, we will strive to assess aesthetic qualities with relevance and surprise being equal. This will not be easy and it must be remembered that in many situations, the opinions shared during subjective tests are likely to have been somewhat confused on this point (Gygli et al. 2013). This is especially true when opinions are solicited from unknown and remote persons, as happens with evaluations on the Internet, for example.
Figure I.2Amazement is another mechanism that leads us to pay particular attention to an image. We then strive to clearly identify the specific elements in the image that fall beyond the known schemas of our representation of the universe.
Pleasure versus arousal
The three motivations behind our attention when we study a photo, namely its beauty, our interest in the subject of the photo, and the surprise it may produce in us, are the result of a long, careful and well-reasoned observation. However, it is also possible to study the immediate effects on ourselves following a very brief viewing of a photo. We therefore seek to identify the most elementary biological effects, which involve only the most basic physiological activation and not the elaborate functioning of cognition and reasoning. This is done by presenting a photo to an observer for a fraction of a second and measuring a few physiological indicators that reflect two independent emotional reflexes: pleasure and arousal8. These experiments were first carried out by psychologists at the National Institute of Health (USA) and were then repeated by many other authors. This experiment was carried out by creating a database with a thousand reference photographs, the IAPS database (International Affective Picture System9) (Lang et al. 1999). They then conducted psychophysiological experiments that made it possible to place each image along a pleasure graph as a function of the arousal, revealing two clear orientations, which they named the direction of appetitive motivation and the direction of defensive motivation (Figure I.4).
Figure I.3Both these photographs are indisputably remarkable for their aesthetic qualities. Photograph (a) (Portrait of Jean Cocteau by Irving Penn, 1948) holds our attention because we recognize a famous man (“universal interest” stimulus). Photograph (b) (Bruxelles by Henri Cartier-Bresson, 1932) holds us by making us wonder “what are they looking at?” (“surprise” stimulus)
A photo that was selected for its capacity to surprise may evoke pleasure or arousal. A photo selected for the interest it arouses in us is selected especially for pleasure, and even more so for a photo distinguished by its beauty, and we will see that even today, this pleasure is a commonly accepted determinant of Beauty. However, literature from the 18th and 19th centuries would sometimes place the Sublime10 above the beautiful. The Sublime had a visceral component of fear (and thus, arousal). The sublime is not commonly used in this sense today and is most often only used as a superlative for beauty.
Figure I.4The affective space defined by Lang et al.
COMMENT ON FIGURE I.4.– The photos from the IPAS database (Lang et al. 1999) are distributed along two axes, one is arousal, which has only positive valence, and the other is pleasure, which has both positive and negative valence. A few remarks isolate specific themes and make it possible to judge the gradation of the affects along the two (hypothetical) axes: appetitive motivation and defensive motivation (Schupp et al. 2004).
Art, Beauty and Aesthetics – how are they related?
Let us now turn to only those images that we judge to be beautiful and let us strive to set aside the more complex motivations that could lead to us being attracted by these images. This text aims to highlight objective elements that would allow us to explain why we attribute this adjective to a photo. However, in order to do this, we must first share a few definitions to give shape to a vocabulary that is often quite fluid.
Aesthetics11 is the science whose aim is to research and determine the characteristics of beauty in works of nature or art (Académie Française, 1835).
Beauty is the quality of that which is beautiful [. . . ]. It is the characteristic of that which evokes admiration and emotion through its forms, proportions, rhythms and harmonies.
Today, we often prefer definitions that place more of an emphasis on the origin of the stimulus as well as the form taken by its effects: “Beauty is the characteristic of a person or an object that makes them pleasurable or satisfying to perceive”. Or, as a poet expressed more lyrically, “Beauty is nothing but the promise of happiness”12.
Art is the range of activities that humans carry out that lead to the production of artifacts, with the objective that they will be appreciated for their beauty or emotional power.
Bringing together these three terms, Aesthetic, Beauty and Art, it can be seen that they cover our field of study. And as each of them has produced abundant literature, it would be tempting to use all three. However, we must be careful with the third term, “Art”. While, for many centuries, the objective of Art tended largely toward the search for Beauty13, its evolution from the 19th century onward must be studied with some circumspection. In particular, we must consider the “fracture in the 20th century” as a definitive change in paradigm. It is quite clear that the second term in the above definition (“their emotional power”) has become the central question in art, superseding “Beauty”14. Looking back, it is easy to find the foundations for this preoccupation well before the Surrealists, as many eminently artistic works (Bosch, Chardin, Goya, etc.) do not claim to be primarily “beautiful” (Figure I.5).
Figure I.5Art that is not very aesthetic.
COMMENT ON FIGURE I.5.– Many works of art have been designed without the “Beauty” criterion being highlighted. The artist seeks to focus on the “emotional load” of their message. These works “with a message” were very much in the minority in the classical period (they are represented here by J. Bosch, F. Goya and J.S. Chardin). They quickly became the dominant form in the 20th century however (represented here by M. Duchamp, A. Warhol, E. Munch, F. Bacon) and it would be difficult to evaluate artwork from the 20th century based only on their aesthetic qualities.
Given this split in the very objectives of Art, we must be very careful when relying on texts in our quest of the Beautiful. And while this is especially true for paintings, it is also the case in photography, which also rightly aspires to be classified as an Art15. We must be very prudent when following the recommendations of photographers, especially the most celebrated, when their advice does not clearly separate the intention of arriving at a greater aesthetic from that of producing a greater emotional impact. We will also see that a clear definition for what Art is in society (see, for example, Danto (1992); Danto and Goehr (2014)) is particularly useful in answering two frequently heard questions that spark off heated debates on contemporary art:
– Is a perfect copy of the Mona Lisa as beautiful as the original16?
– Is Warhol’s Campbell Soup Can beautiful? Is it more beautiful than the can found in your local grocery store? What makes it a work of art?
Our work
We must first explain why we are taking up this subject today. The opening lines of this book reminded us how abundant images are in society today. They described the complexity of the necessary task of sorting through the mass of photographs and the role played by aesthetics, along with other emotions, interest and surprise, in carrying out this winnowing. We must therefore take into account the progress that has been made in understanding how we judge the aesthetic quality of a photograph and the attempts made in using information processing tools to facilitate this task. Therefore, in 2018, for the first time ever, an international photography contest was judged in parallel by two juries: one composed of expert photographers and the other consisting of a computer running an artificial intelligence program17.
It appears that these two aspects, understanding the mechanisms involved in humans, as well as the development of assistance tools, go together. At least, this was the deeply held conviction that motivated us to launch this study. However, even as this book was progressing (i.e. from September 2016 to 2020), there was a constant stream of results that seemed to contradict this idea: deep neural networks, even though in their nascent stage, ignore the need to understand the relation between the observer and the photo, but still lead to astonishingly good judgments of the aesthetic qualities of the photo. While they are still nowhere near replacing an expert (human) eye as of now, they are well ahead of the other tools developed earlier. This will be discussed at length later in Chapter 8.
Thus, our subject continued to evolve even as we worked on it. We have fewer explanations than we had hoped for initially, but we can give more detailed reports of the procedure undertaken, report on the evolution in the approaches and compare performances and highlight advances and weaknesses.
Our work can be divided into two significant parts.
In the first part, we attempt to give an account of the work undertaken in different fields of knowledge to identify the mechanisms underlying aesthetic perception.
We begin with the vast corpus of philosophical literature, which we have tried to map in a highly schematic manner in Chapter 1. We have tried to simplify the many texts that have attempted, over 25 centuries, to find certain universal rules. The guiding principle we have used is the contrast between the objectivist and subjectivist perception of Beauty. We then examine what biologists have to say (focusing especially on neurologists) about the mechanisms of aesthetic perception. The development of modern brain-imaging techniques has transformed what we knew of the mapping of areas involved in developing emotions and judgments. Work that emerged from the field of art, today grouped under the term “neuroaesthetics” is then presented in Chapter 2. In the 150 years of its existence, photography has also developed a large corpus of recommendations and teachings, often born out of the practice of shooting a photo, but also often extended through experiments and scientific processes based on knowledge gained from experimental psychology. These results, which are often tracked and verified by image processing experts, are presented in great detail in Chapter 3. In the second section, we will then look at how these are used today by algorithms.
In this second section, we examine the work that uses tools born from mathematics and computer science to measure beauty. In Chapter 4, we begin by presenting the (often quite old) techniques that propose a closed form of beauty (like that proposed by G. Birkhoff), which does not involve a stage of learning. Although quite rare, this research is ongoing even today. The next three chapters are dedicated to current methods that have emerged from artificial intelligence. We start with a broad study of the context that makes these processes possible, especially image databases available in the network, and then the expertise that was used to bring in the essential step of machine learning (Chapter 5). Chapter 6 details the first generation of research carried out using the detection of primitives that are assumed to express the aesthetic quality and their classification using machine learning techniques. This research came up in the late 1990s. The following chapter covers more current work, essentially carried out since 2014, using deep neural networks and doing away with any human expertise once an annotated database is set up (Chapter 7). The conclusion to these information science methods reviews their results (Chapter 8), highlighting current directions being explored and presenting future perspectives that seem to be revealed. It is tempting to conclude that the immense progress will be fruitful very soon, but it is just as easy to demonstrate the futility of the efforts undertaken given how naive and inherently limited these results are.
We must of course provide a global conclusion to this book, by putting into perspective the contributions from all the chapters: philosophy, neurology, experimental and social psychology, image processing and artificial intelligence. Unfortunately, we could not succeed in bringing together the various sciences that have worked on the judgment of Beauty. We can, at most, reveal a few key problems that hinder our understanding, related to the deepest visual perception pathways by bringing in the chronology and causality of activations of various brain regions. A better understanding of these mechanisms could result in models that could then be translated into semiconductors.
Finally, four appendices complete the book.
The first complements Chapter 1 by tracing aesthetics from the ancient Greeks up to the Enlightenment, leading to the apogee of what is today called the classical period. This is done with a directive reading of philosophers who fit into the framework of our project of moving from objectivism to subjectivism.
Then, over the next three appendices, we look at aesthetics that followed a different path from that of the West: we travel to China, Persia (in the broader, historical sense) and to Japan18. It seems evident that these aesthetics cannot be served by the approaches used by artificial intelligence as it is today, without entirely redesigning its foundations, which nobody seems interested in doing at present.
Notes
1
It is estimated that 7,400 billion photos have been archived to date, with 1,400 billion being taken in 2020 alone (Carrington, D. (2020). How many photos will be taken in 2020?.
life in focus
:
https://focus.mylio.com/tech-today/how-many-photos-will-be-takenin
).
2
Or at least that is what we assume, and this could be a risky hypothesis and a weak point in this text.
3
Interestingness
– many authors also use this term to refer to aesthetic attraction (Gygli et al. 2013). It can also be used in a very specific sense in the case of social networks, where it measures any form of interest, independently of the causes that are brought into play (Dhar et al. 2011).
4
Memorability
differentiates between images based on a criterion apart from their immediate impact on the observer. It reflects the capacity of an image to enter long-term memory, which is where memories are stored in our mind in a lasting manner. Although this is a universal quality, it is assessed at the individual level (Khosla et al. 2012; Kim et al. 2013).
5
Unusualness
refers to that property of images that is characterized by producing surprise: to this effect, the concerned images are distinguishable from the other images proposed, especially in the case of animated sequences. This property is specific to a given context and may vanish in a different context (Schuster et al. 2010).
6
Popularity
is the term used to characterize images posted on social media. Popularity reflects an interest shared by a large number of people, based on criteria that may be very variable (Amengual et al. 2015).
7
It is especially interesting to see how retrospective collections put together regularly by professional journals, with titles like, “The 100 most beautiful photos of the year”, largely combine photos with exceptional aesthetics and those that evoke exceptional events from sports, politics or world affairs, which have been selected largely for their qualities of being remembered .
8
Arousal: this term, which we have already seen, refers to a person coming out of a state of indifference.
9
A competitor to the IAPS database, OASIS, was created with the same objective by Harvard University (Kurdi et al. 2017). It also offered the advantage of being open access.
10
The Sublime is the subject of Edmund Burke’s first philosophical text (in 1757):
A philosophical enquiry into the origin of our ideas of the Sublime and Beautiful
. The Sublime was soon at the heart of many literary debates across Europe: it brought in the lyricism of enthusiasm to counterbalance good measure and harmony. It was discussed at length by Immanuel Kant in 1790 in his
Critique of Judgement
(Kant 2015) where he writes, in particular: “The Beautiful prepares us to love disinterestedly something, even nature itself; the Sublime prepares us to esteem something highly even in opposition to our own (sensible) interest”. And he added, “The satisfaction in the Sublime of nature is then only
negative
(whilst that in the Beautiful is
positive
)”. Byron waxes lyrical on the Sublime during his travels in a storm in narrow Alpine valleys. We find, in Saint Girons (2005) a historical study of the sublime. Our text has very few references to the sublime, apart from the historical overview of the concept of beauty, as in modern usage it is not separate from extreme beauty.
11
In reality, there are three different meanings for the word aesthetic. We will use here only the definition given above. However, the Greek origin expresses the general sense of perception, a meaning that is rarely used today. The more common meaning is the technical one that designates the choices made by an artist in producing their creation:
Fritz Lang’s aesthetic
.
12
Henri Beyle, writing under the pen-name Stendhal,
De l’Amour
(Of Love), Chapter XVII.
13
Our observations here are restricted to Western art, born out of the Greek school of art. Art from other societies, especially those from the East, was born out of other aims and intentions. Appendices 2 to 4, therefore, offer short studies of the conception of Aesthetics in ancient China, in Persian miniatures, and in Japan.
14
We will speak here of the stated intentions of the movements that largely contributed to art in the 20th century: Dadaism, Surrealism, Situationism, CoBrA, Fluxus, Pop Art, Guerilla Girls, Punk, Kitsch, etc.
15
That photography can be classified as art is well-accepted among art lovers today. A survey by the
Beaux Arts Magazine
in October 2017 showed that when asked, “Using what form of expression are artists most successful in capturing beauty?” the photograph was ranked on top by 32% of French respondents, while the painting received the top vote only in 31% of responses, and the sculpture in 15%. We have come quite a long way from the observation made by Bourdieu (1965) in the middle of the 20th century, calling photography in France, “a middle-brow art”, not only for its quality of production, but also in terms of the esteem it received.
16
This question was long-debated by G.T. Fechner over the Dresden Madonna (see Vidal (2011)). Contrary to the position given in Danto (1992), for example, which presents a consensus that is widely held today, Fechner supported the close dependence between
Schönheit
(beauty) and
Echtheit
(authenticity), denying any beauty to any copy, however perfect.
17
This was the
SPARK, a Renaissance
photographic contest conducted by Huawei, built around the company’s P20 Pro cellphone model, which had an embedded algorithm for the aesthetic evaluation of a picture. The algorithm uses a neural network trained with over 4 million images. This is the algorithm that was in charge of judging the contest, awarding each image a score between 0 and 100. The photographs that got the best scores are available at
https://consumer.huawei.com/it/campaign/sparkarenaissance/prizes/
(August 2020). There was very little information available comparing the judgments of the human jury and the algorithm. Later editions of the contest do not seem to have been judged by an algorithm.
18
We chose these countries as there is a large amount of written material that discusses the aesthetic choices that governed their art. While other societies also constructed very coherent systems of “artistic creation” (Australia, Africa, the Americas, etc.), it is more difficult to find primary texts that specify the rules and means.
1The Legacy of Philosophers
Everyone reasons about the beautiful: it is admired in the works of nature: it is demanded in the productions of the Arts: at each moment its quality is conferred or denied; however, if one were to ask men with the surest and most exquisite taste what is its origin, its nature, the precise notion of it, its true idea, its exact definition; whether it is something absolute or relative; whether there is an essential, eternal, unchanging beautiful that would be the rule and the model for a subaltern beautiful; or whether beauty is like fashions: one would immediately see divided opinions; some declare their ignorance, while others fall into skepticism. How is it that almost all men agree that there is a beautiful; that so many among them feel strongly where it lies, yet so few know what it is?
Denis DIDEROT (1751)
The importance that philosophers have accorded the idea of Beauty, from Antiquity onwards, can be seen in the abundance of major writings arising from this concept. The “theoretical works” of Socrates, Plato and Aristotle, based on the “practical works” of Apelles, Phidias and Praxiteles, provided a general framework for reflection that traveled from Greece to the Roman world and, through Medieval times, finally arrived into the Renaissance. In the Renaissance, this framework was accorded a pre-eminent place in a particularly vibrant artistic and intellectual life. In the century of the Enlightenment, the world of ideas often carried the day during heated disputes (Boileau, Locke, Diderot, etc.) but without being really questioned by the world of the Arts (Tatarkiewicz 1970). Toward the end of the 18th century and in the 19th century, no great philosopher was left out of the debate: Kant, Hegel, Schopenhauer, Nietzsche and so on. All of them produced treatises on this subject and these are essential books in our libraries today, while artists were carrying out a Copernican revolution in how they designed their art, emblematically represented by the Romantics in literature and the Impressionists in painting. In the field of Art, the 20th century dawned on an indescribable battlefield with confrontations between theoreticians, each fervently entrenched in their positions: Heidegger, Adorno, Wittgenstein, Deleuze, Derrida, Foucault, Eco, Danto, etc.1, while the world of artwork was given over to a rare anarchy2.
Of course, every contribution cannot be faithfully reported here. We will present an extremely simplified version (which is therefore highly open to criticism), with the aim of contextualizing the central question of our study and placing certain terms in perspective. Thus, for our purposes, we will highlight two, antithetical points of view of aesthetic thought: the “objectivist” approach, which attributes the merit of beauty only to the object that is beautiful, and the “subjectivist” approach which, on the contrary, locates beauty in the “eye of the beholder”3. We will show how objectivism has given way to subjectivism over the last few centuries, and then show how both of these approaches have resulted, for better or for worse, in a hybridized approach. Above all, we will demonstrate the implications that both these theories have on our project of using computerized processes to determine the beauty of a photograph, and we will look at why these approaches are inevitably present in the expectations from software developed today. We are inclined toward placing a greater emphasis on the objectivist approach as it appears so that all processes that seek to replace human judgment by an automated decision-making process start from an objectivist point of view, even if some of them have been led to adopt a subjectivist viewpoint as they developed.
First of all, let us introduce the question of classifying approaches into two opposing families and to do this we must again quote Diderot, who clearly explained the terms in the Encyclopédie: “It is evident that Saint Augustin had gone much further in his research into the beautiful than the Leibnizian philosopher4: the latter seems to claim first that something is beautiful because it pleases us; when it only pleases us because it is beautiful; as Plato and Saint Augustine have noted quite well”. Does Beauty give pleasure? Or does pleasure create Beauty? Diderot has made his choice, referring to works by Ancient philosophers. But there are indeed two ways of studying the problem, as we will see below. For example, Luc Ferry has penned the exact counterpoint to Diderot, “it is no longer because the object is intrinsically beautiful that it pleases but, rather, we can go so far as to say that it is because it provides a certain type of pleasure that we call it beautiful” (Ferry 1990, p. 25).
Equipped with both these philosophical frameworks, we will turn to more recent results from neurobiology to see whether they make it possible to choose between these approaches or to validate them through brain imaging experiments. We will also examine how compatible they are with results from studies carried out for over a century in experimental psychology, psychosociology and sociology. We hope to then have a collection of work that will enable us to interpret the processes adopted by automated approaches to determine the beauty of photographs.
1.1. The objectivist approach
1.1.1. The source: ancient Greece
The objectivist approach5 was formalized in Ancient Greece, that of Plato6 and Socrates7, and which Aristotle8 also adopted, by humanizing it a little. This was the approach that was bequeathed to the West through the Romans, re-adopted by Augustine of Hippo9 who adapted it to Christianity. It then blossomed in the Renaissance, flourishing in the Classical period and remained dominant until the middle of the 19th century in the world of art, if not in philosophy.
The objectivist approach declares that Beauty is a quality of the concerned person, object or music. This Beauty is the reflection, in the observed object, of the universal properties of harmony in proportion and shape, properties that we do not completely understand, but which exist prior to our observation and influence our judgment.10 For this reason, Beauty is perceived equally by any observer11 at any time and any place. Thus, according to the most reputed art historians and experts, the Venus de Milo, the Apollo Belvedere, and Michelangelo’s David, among others, are endowed with these eternal qualities that demand recognition from humanity. The rules established in Athens in the 5th century BCE, which were adopted into the field of perception, reside in the harmony of form12 and proportion, especially the consonance of the whole and its parts13. The simplest proportions are the most pleasing (1/2, 1/3, 3/4, etc.), but the eye appreciates subtler fractions in more complex constructions, like in architecture. The echo of proportions on various scales and in different parts of the object is important and contributes to the “symmetry”14. All Greek art is designed using a highly visible mathematical framework, which represents the omnipresence of numbers in Athenian cosmogony.
However, while absolute beauty is the business of the gods, the rules of harmony are essentially inherited from nature, especially the proportions of the human body. Nature, a reflection of divine constructions, is the primordial guide of aesthetics. All authors preach the necessary simplicity of works, which must be comprehensible at first glance.
On the other hand, in the field of colors, many ancient rules have been lost and most texts that could have testified to the tastes of the Ancient Greeks have disappeared. We know very little about chromatic harmonies that seem to use a range from black to white, harmonically divided by red and yellow.
Greek rules were stated through “canon”, which is applied quite rigidly to shapes, lines and harmonies. The canons limit artistic creation and innovation was not considered a virtue in artwork. Further, in Athens, the artist was an artisan before being an artist, and was not really a creator, except in the noblest art of the Tragedy or unless he attained perfection, as did Praxiteles, Polycletus or Zeuxis.
The objectivist hypothesis fully justifies the attempt to determine the aesthetic qualities of an object without using an observer, since these qualities are immanent and are expressed with or without an observer. This is further legitimized by the fact that the rules of aesthetics are universal and largely independent of the form of the artifact: painting, music, architecture, sculpture. Indeed, this was how Greek thinkers of the Hellenistic period had conceived of these15.
1.1.2. After Greece
Rome adopted the Hellenistic aesthetic, and while this then became more flexible, naturalized and immersed in society, the whole of the Mediterranean Basin was given over to the rules of Antiquity. Rome insisted on Greek statuary and architecture serving as ideals and the canon as rules. Vitruvius’ text On Architecture16, translated from the late 1st century BCE, transmitted these teachings to us, the original Greek sources having disappeared.
The Roman Empire, however, was large, and between the 6th and 7th century CE, it split into two. The Eastern influences from Byzantine led it away from the Athenian aesthetic toward arts that were more fixed both in their inspiration and their forms. The human gave way to the spiritual. The canons were fixed with abstract conventions, governing the style, materials, forms, expressions and colors. Nature retreated and ornaments appeared, symbols with codes reflecting mysticism and religion. This branch of aesthetics split away from the Athenian line and never returned toward this significant Western school of art.
On the other hand, the Western Roman Empire seized upon the Athenian legacy. In North Africa, Augustine of Hippo, brought up on the Hellenistic aesthetic, declared that the Beauty of Antiquity was objective and universal and could express the divine ideal: simplicity, harmony, a respect for proportions. The beauty of Olympus was bestowed upon the Temple in Jerusalem and the Church in Rome. The universality of Beauty is because of divine transcendence, a reflection of the glory of one God, as it reflected the glory of the pantheon. Christianity, shaped by these teachings, would carry Plato and Aristotle’s precepts across Medieval times, through the copies made by monks and the workshops of artists. In parallel to these, the same teachings were taught along the Tigris and Euphrates and in the distant Samanid empire, where poetry, cosmology and algebra were discussed and debated. Beyond the Byzantine Empire, this ancient tradition was alive and flourishing and would be transmitted, in turn, to Bologna, Montpellier, Coimbra and Paris at the start of the new millennium.
At the turn of the millennium, the year 1000, while aesthetic rules had not evolved very much, Athenian philosophy was reconsidered, especially with regard to the relationship between vision and perception. English and Scottish thinkers began the empirical revolution, which preached the end of the innate and the pre-eminence of cognition in consciousness and emotion. Beauty, like Goodness, was not bestowed upon one at birth, but could be acquired through experience and practice, by following an example. There were still a few centuries to go before the reflections of Ockham, Locke, Hutcheson and then Hume would introduce sceptical relativism (we can refer, among others, to Ferry (1990) and Tatarkiewicz (1970) for the impact these thinkers had on the concept of aesthetics).
At the same time, across the Channel, Descartes17 and Spinoza would state the dominance of reason, the separation between Mind and Body which allows the development of the individual, which was the only reference for the mind: “cogito ergo sum”. We now enter a complex period in aesthetics where the foundations of objectivism were openly undermined, but the walls held strong and a scaffolding of arguments were put up to support the conclusions that society wanted to hear: beauty is universal, timeless and can be immediately perceived by all. For some people, this was a part of God, placed in us so we could judge Beauty and Justice equally, alongside our free will. For some others, this was a pleasing goal of Nature, which ensured harmony in judgment. Others still saw this as the result of an as-yet-unknown biological organ that “perceived” beauty as the eye-perceived light. This was a period where scholarly texts were entirely given over to aesthetics18: Jean-Baptiste Du Bos, Charles Batteux, Dominique Bouhours, Jean-Pierre de Crousaz, William Hogarth, etc.
This was also the time when debates between luminaries arose again in society: the Quarrel of the Ancients and the Moderns was a debate between ideas that defended Classicism and ideas that fought for “an aesthetic of sensibility” and all works of art turned into a battlefield between these two currents: poetry, music, painting, decoration and architecture. Faced with the rise of the individual and subjectivity, the objectivist position became weaker and weaker: Johann Joachim Winckelmann was one of the last ramparts to stand firm19, while in the field of philosophy the arguments constructed by Immanuel Kant would form the modern foundations of aesthetics, the way in which the field is still largely discussed today (Schaper 1964; Aquila 1970; Crowther 1976; Hopkins 2001). Kant is held to be the primary reference on subjectivism20; however, his vision of Beauty is profoundly influenced by objectivist precepts. In the following paragraphs, we will explain why we consider Kant to be the last of the objectivists and the first of the subjectivists.
1.1.3. Kant and modern aesthetics
The complexity of the universal thinking on aesthetics is seen in Kant’s oeuvre, especially in his Critique of Judgement (Kant 2015), which came out shortly after his Critique of Pure Reason and uses many results from the latter.
For Kant, Beauty is the product of the “judgment of taste” and thus arises from judgment and not from perception. Today we would say that it is a cognitive act, and not a perception (“red”, for example, is purely a product of perception). However, Kant also differentiates between several kinds of cognitive judgments. “Beauty”, according to Kant, is derived from perception, like “square” would be, for instance, and both are products of the mind. Nonetheless, “square” is not a judgment of taste, it is a reasoned result of the application of “concepts”, while a judgment of taste does not require any concept that would “explain” the judgment. It is simply a product of the pleasure experienced during the observation. “Beautiful”, “Good” and “Fair” all share this property of being products of judgment of taste. On the other hand, “red” cannot be disputed. If 20 people tell me that this object is red, but I only see brown, then I would have to question my visual system. If 20 people tell me that a silhouette seen at a distance is a dog, while I think it was a wolf, we would be able to talk about the cues and reasoning (“concepts”) that led me to the decision and I would probably agree with their opinion or they might change theirs. However, if I judge a painting to be ugly while 20 people find it beautiful, Kant believes that other people’s opinions would not change mine, since, “that a thing has pleased others could never serve as the basis of an aesthetical judgement” (Kant 2015). It can, at most, make me a little less confident of my opinion21.
Is aesthetic judgment objective or subjective according to Kant? Even Kant (2015) finds this a tricky question. He writes: “The green color of the meadows belongs to objective sensation, as a perception of an object of sense; the pleasantness of this belongs to subjective sensation”. However, he specifies that, “[...] the satisfaction presupposes not the mere judgement about it, but the relation of its existence to my state, so far as this is affected by such an Object. Hence we do not merely say of the pleasant, it pleases; but, it gratifies. I give to it no mere approval, but inclination is aroused by it...”. He supplements this, a little further, by saying, “the judgement of taste is merely contemplative
