139,99 €
Mehr erfahren.
- Herausgeber: John Wiley & Sons
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
Digital practices are shaped by graphical representations that appear on the computer screen, which is the principal surface for designing, visualizing, and interacting with digital information. Before any digital image or graphical interface is rendered on the screen there is a series of layers that affect its visual properties. To discover such processes it is necessary to investigate software applications, graphical user interfaces, programming languages and code, algorithms, data structures, and data types in their relationship with graphical outcomes and design possibilities. This book studies interfaces as images and images as interfaces. It offers a comprehensible framework to study graphical representations of visual information. It explores the relationship between visual information and its graphical supports, taking into account contributions from fields of visual computing. Graphical supports are considered as material but also as formal aspects underlying the representation of digital images on the digital screen.
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 375
Veröffentlichungsjahr: 2017
Ähnliche
Table of Contents
Cover
Title
Copyright
Preface
Introduction
1 Describing Images
1.1. Light, visual perception and visual imagery
1.2. Visual media
1.3. Visual interfaces
1.4. Visual methods
2 Describing Graphical Information
2.1. Organizing levels of description
2.2. Fundamental signs of visual information
2.3. Visual information as texts
2.4. Objectual materiality of visual information
3 Practicing Image-Interfaces
3.1. Mise-en-interface
3.2. Studying graphical interface features
3.3. 1970s: Xerox PARC years
3.4. 1984–1996: Classic Mac OS years
3.5. 1995–2005: Windows 95, 98 and XP years
3.6. 2005–2017: Free software years and web as platform
3.7. Interface logics
4 Designing Image-Interfaces
4.1. Precursors to data visualization
4.2. Data visualization
5 Prototyping Image-Interfaces
5.1. Scripting software applications
5.2. Integrating data visualizations
5.3. Integrating media visualizations
5.4. Explorations in media visualization
Conclusion
Appendices
Appendix 1
Appendix 2
Appendix 3
Bibliography
Index
End User License Agreement
List of Tables
2 Describing Graphical Information
Table 2.1. Levels of description. Semiotic trajectory of expression with an adaptation to digital images
Table 2.2. Numerical notations: decimal, hexadecimal and binary
Table 2.3. Color notations: W3C name, RGB, HSB, hexadecimal, binary code
Table 2.4. Common operator symbols and their description in the JavaScript language
Table 2.5. From mathematical notation to programming syntax, adapted from Math as code by Matt DesLauriers
9
. A reference of mathematical symbols for UTF-8 formatting is also available online
10
Table 2.6. List of visual attributes in CSS, SVG and OpenGL
Table 2.7. HTML forms, input types, jQuery UI widgets and interaction methods
Table 2.8. Interface patterns, from [TID 11]
Table 2.9. Data keywords in the OBJ file format
41
Table 2.10. Generic structure of JPEG files
4 Designing Image-Interfaces
Table 4.1. Some precursors to data visualization and diagrammatic reasoning
Table 4.2. Dimensionality reduction methods
5 Prototyping Image-Interfaces
Table 5.1. newLayers.js. JavaScript code to be run with Photoshop
Table 5.2. mayaFractal.py. Python script to be run with Maya
Table 5.3. ImageMeasure-RGB. Macro script for extracting RGB colors
Table 5.4. MotionStructure-v2.imj script
Appendix 1
Table A1.1. Summary of commercial software applications
Table A1.2. Summary of open-source software applications
Table A1.3. Summary of free commercial software
Appendix 2
Table A2.1. Summary of web-based tools
Table A2.2. Summary of directories for web tools
Table A2.3. Summary of JavaScript, CSS and HTML libraries
Appendix 3
Table A3.1. Summary of online exploratory data visualization projects
List of Illustrations
1 Describing Images
Figure 1.1. Typical composition techniques
Figure 1.2. Examples of proto-iconical images (visual jokes)
Figure 1.3. Visual variables according to Jacques Bertin
2 Describing Graphical Information
Figure 2.1. Data structures for image processing
Figure 2.2. Data structures: types of lists and trees
Figure 2.3. Data structures for image synthesis
Figure 2.4. Euclid’s algorithm and flowchart representation [KNU 68, pp. 2–3]
Figure 2.5. Geometrical transformations (created with Adobe Illustrator filters)
Figure 2.6. Image filtering. FFT images were produced with ImageJ, all other with Adobe Photoshop
Figure 2.7. Color segmentation. Images were produced with Color Inspector 3D for ImageJ. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 2.8. Network graph of main tokens, statements and types in Java, Python, C++ and JavaScript programming languages. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 2.9. Different types of control widgets as they are rendered on the computer screen, inspired from Qt Quick Controls module. The graphic style has been simplified. It may actually change in different platforms
Figure 2.10. Common types of grids
Figure 2.11. Logical operations (transfer modes)
Figure 2.12. Barrel and pincushion effects in Nik Collection
3 Practicing Image-Interfaces
Figure 3.1. Xerox Alto interfaces
Figure 3.2. SuperPaint and Trillium interfaces. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 3.3. Menus and controls from Macintosh User Interface Guidelines
Figure 3.4. Classic Mac OS interfaces. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 3.5. Windows 95, 98, XP interfaces. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 3.6. Free software interfaces. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 3.7. Sketches for mapping interface logics
4 Designing Image-Interfaces
Figure 4.1. Peirce’s types of signs
Figure 4.2. Geometrical representations of dimensionality reduction [DON 03]. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 4.3. Chart types
Figure 4.4. Plot types
Figure 4.5. Matrix types
Figure 4.6. Network layouts
Figure 4.7. Geometrical sets
Figure 4.8. Variations and combined types
Figure 4.9. seealso.org. A volunteer-run design studio created in 2012 that curates data visualizations made with Wikipedia data. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 4.10. UML activity diagram of OpenGL graphics
17
Figure 4.11. Circuit layout
Figure 4.12. Example of MRI visualization made with OsiriX
Figure 4.13. Text visualization types
Figure 4.14. Visualization of Ulysses by James Joyce using Voyant Tools For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 4.15. Media visualization types
5 Prototyping Image-Interfaces
Figure 5.1. Generated image with Photoshop using JavaScript. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.2. Orthogonal views of the generated layers. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.3. Generated model using Python with Maya. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.4. MediaViz, 2010. Screenshot of Flash movie and static image. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.5. Map of digital arts. 53330 nodes extracted from Wikipedia. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.6. Visualization of Nirvana videos. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.7. Network media visualization of Robert Fisher. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.8. The NirViz app interface. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.9. Visualizations of Processing 2. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.10. Graphical interventions in digital poetry
Figure 5.11. Interactive image mosaic of 2000 rock album covers. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.12. Image plot of 2000 rock album covers. X = years; Y = hue. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.13. Experimental radial plot of images. r2 = x2 + y2 ; x = r cos(θ); y = r sin(θ). For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.14. Experimental radial plot of images. r2 = x2 + y2 ; x = r cos(θ) * cos(ι); y = r sin(θ) * cos(ι). For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.15. Sketches of polar plot variations. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.16. Adapting HTML positions to media visualization. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.17. 609 Marvel comics covers by season, published between 2000 and 2015 (created by Blumenfeld, Mauchin and Lenoir). For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.18. 120 dress photos from Oscar awards between 2010 and 2016 (created by Corviole, Legrand, Magny, Molina and Nguer). For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.19. D3.js force-directed graph with images. Created by J. Caignard. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.20. Parallel coordinate visualization of multiple dimensions
Figure 5.21. Image plot of PCA factors. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.22. Polar plot of Rothko images of paintings. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.23. Polar plots of Nirvana videography. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.24. 3D plots of Rothko images. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.25. Interactive web-based 3D media visualization. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.26. Interactive web-based 3D media visualization of variation SV1 of algorithm Polar HSB
Figure 5.27. Image generator based on visual attributes and composition rules. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Figure 5.28. Overall volumetric media visualization workflow
Figure 5.29. Motion structure from the title sequence of Game of Thrones
Figure 5.30. 3D print motion structure from the title sequence of Game of Thrones
Figure 5.31. Volumetric visualization of Bill Viola’s “Acceptance”
Figure 5.32. Volumetric visualization of Charles Csuri’s “Hummingbird”
Figure 5.33. Volumetric visualization of Peter Wibel’s “Endless Sandwich”
Figure 5.34. Goggle Viz
Conclusion
Figure 1. Disrupting 3D models. Three geometric configurations of the same 3D object. For a color version of the figure, see www.iste.co.uk/reyes/image.zip
Guide
Cover
Table of Contents
Begin Reading
Pages
C1
iii
iv
v
ix
x
xi
xii
xiii
xiv
xv
xvi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
233
234
235
236
237
238
239
241
243
244
245
247
248
249
251
252
253
255
256
257
258
259
260
261
262
263
264
G1
G2
G3
G4
G5
G6
G7
e1
Digital Tools and Uses Set
coordinated byImad Saleh
Volume 3
The Image-Interface
Graphical Supports for Visual Information
Everardo Reyes-Garcia
First published 2017 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentioned address:
ISTE Ltd27-37 St George’s RoadLondon SW19 4EUUK
www.iste.co.uk
John Wiley & Sons, Inc.111 River StreetHoboken, NJ 07030USA
www.wiley.com
© ISTE Ltd 2017The rights of Everardo Reyes-Garcia to be identified as the author of this work have been asserted by him in accordance with the Copyright, Designs and Patents Act 1988.
Library of Congress Control Number: 2017952099
British Library Cataloguing-in-Publication DataA CIP record for this book is available from the British LibraryISBN 978-1-78630-062-1
Preface
This book studies interfaces as images and images as interfaces. By interfaces, I mean specifically graphical user interfaces; the systems which we use to interact with information displayed on an electronic support.
My interest on this topic has grown over the years and it reflects both personal motivations and professional work as a teacher and researcher. As someone who has always been fascinated with technological objects (I remember as a child programming a VCR, discovering how to change the time in an LCD watch and climbing to the rooftop to redirect the TV antenna), I think one of the best ways to approach a domain is to “touch it”, to do things with it, either under special supervision or under our own risks. I hope this book will provide inspiration to the reader in producing his/her own examples and prototypes, either by hand or with software applications or with computer code.
I have had the opportunity to discuss many of the topics included in this book with colleagues and students. I learned a lot about computer graphics and 3D software while serving as a program director of the BA in Animation and Digital Art at Tecnológico de Monterrey in Toluca, Mexico. Later, I could appreciate the intricacies of web design and human–computer interaction for the web when I was appointed as a program director of the MA Interface Design at Université Paris 13. More recently, data culture, data visualization, digital humanities, semiotics, cultural analytics and software studies appear on a daily basis at the heart of my current professional family at Université Paris 8.
I want to extend my thankfulness to colleagues and students around the world who have kindly invited me or deposited their trust in me to collaborate somehow. Friends in Mexico at Tecnológico de Monterrey, UAEMex, Centro Multimedia, CENART, UAM Xochimilco; in USA at Cultural Analytics Lab and UCLA; in Belgium at Université de Liège; in Italy at Università di Torino; in UK at King’s College and Winchester School of Arts; in Sweden at Lund University; in France at médialab Sciences Po Paris, ENS Paris, ENSAD, MSH Paris Nord, IEA Paris, MESHS Lille, Université Paris 3 Sorbonne Nouvelle; in Brazil at UFRJ; in Spain at Universidad de Castilla La Mancha; in Argentina at Universidad Nacional de La Plata and Universidad de Buenos Aires. I hope we all meet together at the same space and time one of these days.
Everardo REYES-GARCIAAugust 2017
Introduction
This book is situated within the period of time that has been influenced by the information age. The time span has developed from the invention of digital computers to the contemporary scene of massive generation and storage of data. From the standpoint of information and communication studies, this period of time is deeply related to electronic media. In this book, we understand data as digital information and electronic media as the means to access and make sense of it.
In our present study, we are interested in a particular modality of electronic media: the graphical mode. Although most of the physical world perceptible to human senses has been simulated and described with electronic media (sounds, smells and tastes can all be quantified and stored as bits and bytes for a latter representation), the vast majority of developments in electronic media have taken place in visual form. We look at the computer screen as the main material support where digital information is represented in order to convey meaning. Nowadays, screens have become pervasive. We see and interact with them in laptop and smartphone displays, in desktop monitors, as beamer projections, in LED displays in elevators, city signage, museum exhibitions, park attractions, airports and public transportation. Screens can be public or private and shared or intimate (such as head-mounted displays).
The kinds of images depicted in computer screens are closely related to the display technologies that underlie them. As computing power increases, for instance, supporting more complex and multiple tasks, larger files or higher resolutions, the look and style of images reflect on those advances. Consider the differences between a 1980s videogame graphics and today’s consoles. Or let us evoke the first computer animations that were used in music video clips to compare them with today’s amateur and blockbuster films. The World Wide Web has also witnessed this evolution. The first images supported by web browsers are not the same in quality and resolution if we consider recent developments in technologies such as CSS3 and WebGL. What we can observe in all these cases is foremost a quest for realism and hyperrealism. Technicians and artists have devoted great efforts to make images look “more real”, i.e. to represent the physical world with more detail, with the highest possible similarity.
But at the same time, there is another kind of image in electronic media. In order to produce those realistic images, creators use the same computer screen in a different manner. They use graphical interfaces. Whether the authoring software is based on user interface elements such as windows, icons, menus, pointers, or uses a command-line console, it is the same rectangular space of the screen that is redrawn to allow the interaction between data and meaning.
We are interested in studying the computer graphical interface as an image. For that matter, we have to take a step back from the representation of “realistic” images. Graphical interfaces are visual supports of information and we look at the way in which the interfaces are constructed to convey meaning. Throughout the history of computing, the community of researchers in human–computer interaction (HCI) has developed models based on cognitive and linguistic capacities in order to describe efficient ways to create graphical interfaces. Our study intends to contribute to the domain by relating interfaces to images. In other words, it is about regarding the visual constituents of interfaces before and at the moment they are identified as interfaces.
To illustrate our purpose, it is possible to list some visual constituents that exist not only in interfaces but also in any other visual form of image. Following the theoretical tradition of visual semiotics, we can talk about plastic categories: colors, forms and textures. For each category, there are smaller units of analysis (colors can be described according to numerical values of saturation, brightness, “redness”, “greenness” and “blueness”). It is interesting to note that, when we describe a picture or an interface, we can establish the relationships between units at the same granularity level or we can reassemble them in other semantic units. Imagine, for this case, that the position of forms in an image motivates the perception of vectors of direction, by means of graphical abstraction and Gestalt principles. Hence, we could now examine “vector” patterns in different images, for example.
Studying interfaces as images means that there is a visual meaning behind the construction of any authoring and exploring environment. Whereas one of the main influences of HCI in everyday interfaces can be perceived through the WIMP paradigm (Windows Icons Menus Pointers), it is necessary to broaden this perspective. Authoring software has a long tradition that builds on reading–writing schemes. The pioneering research on hypertext systems during the 1960s – long before the invention of the World Wide Web – introduced models for nonlinear writing and reading. Among such models, we may cite the relational, the taxonomical and the spatial.
While the first authoring environments were mainly conceived for textual content, the introduction of different types of media came with an explosion of paradigms and visions. With the support for images, video and audio, creators could now explore storytelling in different ways. However, it was necessary to think about new metaphors to signify actions on abstract objects: to link, to import, to animate, to copy, to delete and to paste. Closely related to graphical interfaces, there is also the development of programming languages and programming paradigms. Starting from declarative and functional languages, we then saw the emergence of generative and object-oriented languages. The syntax is, of course, a visual matter. The kind of programming marks, such as dots, commas, brackets and curly brackets, is a way to represent dependencies, inheritances and encapsulations of data. It is not surprising that more recent graphical programming languages were introduced: boxes connected to other objects by means of links and edges.
As it occurs with photographs and paintings, an interface as image can be approached as a diagram, symbol, index or other category of sign. The image of an interface conveys in itself the traces of its production. It is possible to relate the operative system, the authoring software and the complexity or simplicity of functionalities. It is also a symbol of its time: the visions foreseen to be accomplished with electronic media; the ideologies associated with such productions. Interfaces, through language and affordances, are meta-images of enunciation that speak to their users, in a similar instance as mirrors, reflections, windows and depictions of the painter inside the painting.
Today, any user of electronic devices can take photographs, add visual effects, edit them, share them and comment on them. The new user has become a sort of media producer. Graphical interfaces have become easier to use, at least at their basic level. While we can study the evolution and the effect of portable devices in the esthetics of social images, we are more interested in: how do designers produce and distribute data and images? Which functionalities are integrated and which ones are left behind? How are our new images and interfaces shaping the emerging post-digital culture?
Designers in the information age are expected to invent new ways to explore information and to make sense of data in environments that change constantly. As it happened to pioneers of authoring systems, designers need to investigate new metaphors, new platforms, new actions that are possible with digital and networked technologies. In this context, designers and developers have embraced the World Wide Web as a privileged environment to create and distribute their creations. The technological possibilities of the contemporary web, along with the development of computing devices, allow experimenting with and dreaming about such new models. The tradition of multimedia authoring systems converges today with massive quantities of social and historical data. The kind of interfaces that designers produce is related to reading–writing software in the sense that they are interactive, they support explorations, discoveries and insights that might have passed unattended otherwise. Designers are like image craftsmen because they take decisions about visual composition, arrangements and metaphors. Moreover, they must reflect on the potential outcomes of the interface: the kind of data, format and media, the parameters that can be modified, the series of actions that a user can perform, the pertinent vocabulary, language and syntax, etc.
While common users have increasing access to cameras and image editing software, designers have a wide variety of production tools at their hands. Some research is necessary to investigate the kind of software available (desktop, mobile apps and web apps) as well as programming languages, APIs and libraries that handle data and communicate with the ever-changing environments. Although many software and production tools are developed with a particular objective in mind, it is interesting to see that tools can be mixed and remixed. While the material support remains constant (the screen), it is when we pay attention to its configurations that innovations take place.
Thus, where we position our study design will be neither a specific discipline nor a bunch of recipes and ready-made toolkits to add visual effects or improve the visual style of a project. Design will be understood as a way of life. It is more like a reflective thinking posture about our practices in order to produce new kinds of explorations and actions. It is about questioning defined models to propose unexpected schemes. It is about producing meaningful tools that communicate aptly in the context in which they function. It is about going beyond the pre-defined figure of a “user” and to consider her/him/them as “visitors”, “wanderers”, “researchers”, “learners” and “meaning-makers”. It is about taking seriously and responsively the work of tool-making; to remember that through our work, we are also shaping ways of being (as it was suggested by Winograd and Flores [WIN 86, p. ix]).
As we said, screens have become pervasive; they are material supports of representations. But at the same time, the screen is alive; it reconfigures itself as formal support. The computer screen is today’s canvas of painters, designers, researchers and students in information and communication. Interfaces are the pervasive kind of images of the information age.
Contents of the book
Chapter 1 compiles relevant insights forged within the tradition of sciences, humanities and media studies about the study of images. Although this book is not about interpreting cultural images, it does take into account research that pays attention to material and technological descriptions of images. This is important because we will identify levels of analysis and methodologies that in other chapters will be used as strategies to design and produce visual interfaces. In this respect, a brief discussion regarding the notion of interface is also contemplated.
Chapter 2 is devoted to the foundations of graphical information. In this part, we consider images as data and digital information. We look at the technical aspects that define a digital image: from data types and data structures required to render an image on the computer screen to seminal algorithms and image processing techniques.
Chapter 3 adopts a pragmatic perspective towards image-interfaces; it studies how image-interfaces are practiced, observing the different manners in which those elements are put together, notably through software applications. This chapter is organized into different categories according to practices of imaging, for example, image editing, parametric design, and also writing and web environments. As long as we talk about image-interfaces, we are not only concerned with the results of some software manipulation, but also with the graphical elements that accompany the user to achieve such transformations.
Chapter 4 pays special attention to those cases where the image being manipulated acts as the graphical interface. It deals with multidisciplinary fields at the crossroad of several disciplines, such as graphic design, data science, computer science and humanities. The kinds of results that are produced from these perspectives include data visualization, networks, graphs, infographics, cultural analytics and data art.
Chapter 5 presents our own productions and experiments created within the context of image-interface and data visualization. It covers a range of prototypes, screen shots and sample codes for scripting software applications, integrating data and media visualization, and extending the generated images beyond the screen, to “data physicalization” via 3D printing.
The conclusion points to perspectives that still have little or recent exposition to information sciences. The term “visual hacking” is used to make reference to speculative advances in image-interfaces inspired by the non-visible electromagnetic spectrum, non-Euclidean geometry, obfuscated code and exotic data structures, and critical and radical design. Finally, the appendix section summarizes the software applications in the form of tables, which are discussed in Chapter 3. It also includes the tables of JavaScript libraries, web links to data visualizations and tools.
1Describing Images
This chapter explores relevant insights about the study of images that have been forged within the sciences, humanities, and media studies traditions. Although this book is not about interpreting images, it does take into account research that focuses on material and technological descriptions of images. This is important because we will identify levels of analysis and methodology which in other chapters will be used as strategies to design and produce visual interfaces. In this respect, a brief discussion regarding the notion of interface is also contemplated.
1.1. Light, visual perception and visual imagery
Where do images come from? Where do they take place? Before images are printed or displayed on a computer screen, they are physical and psychological phenomena. A brief account of the processes underlying the formation of images will illuminate the perceptual and cognitive approaches that will inform further sections of this book.
On the one hand, visual perception puts attention on the physical and material aspect of vision. It occurs in the eye and its organic layers. The physical explanation of vision starts when light stimulates the retina. From there, different photoreceptor cells, mainly rods and cones, process signals and send information to the region of the brain called the primary visual cortex. On the other hand, visual imagery is related to mental experiences of representation and simulation; it occurs in the brain. Henceforth, the explanation moves to the domain of cognitive sciences. The cortex identifies basic visual features and redirects information according to two pathways: visual properties (shape, color) and spatial movement properties (location and motion).
Vision and imagery cooperate whenever we interact with images depicted on the computer screen. As we will note, we do not only perceive and explore visual constituents of images but also think about different things at the same time: maybe a mental scheme of the interface, a subjective critique of design decisions, a memory evoked by a picture, or even our grocery list or plans for the next weekend.
1.1.1. Physical light
Light is what allows visible sensation. It is worth starting our account on visual media by considering some physical conditions of light as they have been postulated by sciences. This section will be useful to review basic concepts, properties, units of measure, and applications derived from the study of light.
Optics is the branch of physics concerned with the phenomena of light, particularly its propagation and image formation [TAI 13, p. 485]. Broadly speaking, optics has three subdomains, each one describing light differently: geometrical, wave, and quantum optics.
For geometrical optics, light is understood as a series of rays or beams. Here, the object of interest is not really the physical properties of light, but rather the “controlled manipulations of rays by means of refracting and reflecting surfaces” [HEC 05, p. 156]. The light that arrives from the Sun, for example, crosses the atmosphere, which is composed of air molecules that conform the density of the medium. Light particles interact with these molecules at different moments and angles, varying its diffusion: lateral diffusion produces the blue of sky; low diffusion, when the Sun is closer to the horizon, the red-orange of dusk; and, after 18 degrees below the horizon line, the black of night. This optical phenomenon is also referred to as Rayleigh scattering.
When the light hits more solid substances than air, it is said to be refracted and/or reflected. The former specifies the change of direction as the light traverses a substance. The latter occurs when the light is returned or bounced off the surface. More precisely, reflection can be of two kinds: specular (when the reflecting surface is smooth, creating a mirror image) and diffuse (when light bounces off in all directions). In the natural world, both phenomena rarely occur in pure manner; the correct behavior lies somewhere between the two [HEC 05, p. 103].
For our purposes, geometrical optics will be further evoked regarding its applications: it has been an integral part of the development and understanding of optical systems (human’s eye, glasses, magnifying glasses, binoculars, microscopes, camera lenses, and telescopes); it also provides explanations for peculiar optical phenomena in nature (mirages, rainbows, halos, shadows, etc.)1, and it has informed the development of software for computer graphics (such as 3D projection techniques like ray tracing).
For wave optics, light is studied as radiation, that is, as energy transmitted in the form of waves. In this respect, signals can be described as a spectrum consisting of frequencies (times per occurrence measured in Hertz, where 1 Hz equals one oscillation per second) and wavelengths (distance between repetitions of the shape measured in meters). From this perspective, light waves are part of the larger electromagnetic spectrum, which includes other types of radiation: gamma, X-ray, ultraviolet, visible light, infrared, T-ray, microwaves, and radio waves.
Visible radiation can be perceived by the human eye and analyzed according to the visible spectrum. It identifies approximate wavelengths for spectrum colors, going from violet (400–450 nm), blue (450–500 nm), green (500–580 nm), yellow (580–590 nm), orange (590–620 nm) and red (620–700 nm) [TAI 13, p. 635]. The other types of radiation can be detected or generated with special instruments, although they are not always of an optical nature.
Wave optics investigates the superposition of waves mainly through polarization, interference and diffraction. The first takes advantage of the fact that natural light waves oscillate in multiple directions, and therefore it is possible to filter and change the direction of the electromagnetic field. Famous cases where we see applications and types of polarized images are in stereoscopy using 3D glasses, photography lens filters, and liquid crystal displays (LCD). Interference and diffraction use barriers and slits of different shapes (rectangular, circular, single and multiple) to describe how waves move around or change when crossing an opening space in the obstacle. Even though it is not always appropriate, interference considers small number of waves, whereas diffraction deals with a big number [HEC 05, p. 459]. Among the applications, we see some effects in fringes of light, interferograms, speckle textures, and Airy rings.
Finally, for quantum optics, light is studied at the subatomic level where the fundamental particle is the photon. It describes the minimal amount of all electromagnetic radiation and is part of the boson classification, together with the gluon, the Z boson and the W boson. The whole picture of elementary particles includes fermions (quarks and leptons that correspond to matter and anti-matter particles)2. Among the applications of photons in optical technologies and imagery, we cite the diverse varieties of lasers (acronym of Light Amplification by Stimulated Emission of Radiation, introduced in 1958), the multiple techniques used in holography (holograms by reflection, transmission, volume holograms, interferometry holograms, etc.) [HEC 05, p. 640], and ongoing advances in quantum computing.
1.1.2. Visual perception: Gibson’s ecological approach
The ecological approach to visual perception differs from physical studies of light by focusing on the perception of the environment. The approach was initiated by the renowned psychologist James J. Gibson [GIB 86] during the second half of the last century. For him, environment is the physical terrestrial world constituted by substances and surfaces. The latter are the visible faces of the former, inasmuch as substances keep a more or less solid state of matter. In other words, surfaces separate substances. For example, consider your field of view right now. You see different objects, perhaps a chair, a table and a wall. Before calling them by their names, you discern their boundaries and contours because some parts are more or less illuminated; they also have some kind of texture, and they are placed at different positions from your angle of perspective (up, down, in front, behind, aside, etc.).
According to Gibson, what we really put attention to when we perceive the environment are not the properties of the surfaces, objects and places (e.g. their shape, color, composition, size) but their “affordances”, that is, what the object permits one, or not, to do with them. If we come back to the chair, table and wall, the chair affords sitting (but also standing on it, or even grabbing, lifting, pushing, pulling if it is not too heavy or attached to the ground). The table affords placing other objects on top of it, and the wall affords separating spaces, hanging objects, painting or covering with tapestry. An important aspect of affordances is that they are properties taken in reference to the observer; they are not physical or phenomenal [GIB 86, p. 143]. In this respect, the chair, table and wall afford differently to a child, a cat or a fly.
As we can infer, Gibson was really occupied by perception in the natural world. He introduced notions such as “ecological optics”, “ecological information” and “ecological events” that helped him re-contextualize traditional understandings of psychology. The importance of his approach was to study visual images as a kind of knowledge. Because in nature we do not have descriptors, visual perception is an implicit and direct knowledge that coexists with others: metric knowledge (by means of optical instruments that allow one to change scale), explicit knowledge (by means of languages that define and categorize objects and places) and mediated or indirect knowledge (by means of pictures, images and written-on surfaces).
The work of Gibson inspired the development of computational solutions to capture and record visual information. We will come back to this question in the following chapter. For now, let’s review what images are from the ecological approach to visual perception.
In Gibson’s terms, images and pictures are surfaces, which means they exist as part of the world, within the context of other objects and surfaces. They have the property of specifying something other than what they are [GIB 86, p. 273], in the same line as “displays”, which are surfaces shaped to show information for more than just the surface itself (e.g. the surface of porcelain, which could be molded as a figure or as a traditional lithophane from Limoges). In the words of Gibson, images and pictures are “surfaces so treated that they make available an optic array of arrested structures with underlying invariants of structure” [GIB 86, p. 272].
From this standpoint, drawings, paintings and photographs record a moment from the real world experience. The ambient light is divided into component parts, which are then organized into an optic array made of visual angles. Such angles may vary according to the point of view of the observer, but they form relations of inclusion: smaller parts or details can be seen within a larger structure (to illustrate, imagine the hierarchy leafs-trees-forest). The so-called “invariants of structure” are elements common to all points of observation. Furthermore, in the specific case of drawings and paintings, where the artist can add imaginary objects, they are also depicted as surfaces. Although previously visualized and abstracted by information processed by the perceptual system, imaginary objects need to be presented in a visual form, with the available material possibilities.
1.1.3. Visual imagery and cognitive sciences
We said earlier that visual imagery is related to mental experiences of perception and action in the absence of their external expression. Visual imagery is also referred to in literature as mental imagery, imagination, or seeing with our mind’s eye. In this section, we move from the natural world to the realm of the brain and mind. The intention is twofold. First, we want to revise how both dimensions could converge and provide helpful insights regarding creativity and innovation. Second, it is crucial to evoke notions of “representation”, “description” and “simulation” within the context of cognitive sciences because they have played a central role in the development of digital images.
In contrast to the ecological approach to visual perception, the concept of “representation” becomes central to cognition. For Gibson, representation is misleading because it would imply recreating a whole natural scene. From this point of view, it is of course impossible to represent exactly the same physical conditions in space and time of an event, and that is why invariants record some of its characteristics.
However, in cognitive sciences, “representation” is a form of mental imagery. The notion is associated with interpretation and meaning making. In the case of images, we say that they are “iconic representations” [AND 04, p. 37], and this already suggests that there exist other types of representation systems: language, words, music, mathematics, logics, etc.
Following the cognitive science approach called “connectionism”, the information stored in the brain is not in the form of explicit packages. It is rather distributed and loosely connected. Hence, mental images imply being reassembled and connected, instead of loaded directly from the memory [HOF 85, p. 410]. Moreover, the structure seems to be rather dynamic, since the elements of a representation system trigger modifications in different parts or processes of the mind (this is what Marvin Minsky refers to as “agents” [MIN 85, p. 196]).
One of the important aspects of regarding mental imagery for our study will be to consider it as a source of creative thinking. This will be decisive when we will talk about designing innovative graphical interfaces. To take a glance at imagination and simulation is also to think reflectively about our experiences. It is a way to visualize new virtual features that could be later implemented in projects and interactive interfaces.
Minsky has noted that “there is an intimate relationship between how we ‘represent’ what we already know and the generalizations that will seem most plausible” [MIN 85, p. 303]. In another strand, Douglas Hofstadter, when relating the work of artists and scientists, mentioned the example of a musical interpreter who establishes a multidimensional cognitive structure that is a mental representation of the song, fed by many other different mental structures related to previous experiences [HOF 85, p. 653]. The meaning of the song emerges in the process. Once again we observe the dynamic and complex situation of elements of the representation system, acting both as triggers and substantial supports of signification.
In more recent studies, Benjamin Bergen has introduced the term “embodied simulation”. He proposes that “maybe we understand language by simulating in our minds what it would be like to experience the things that the language describes” [BER 12, p. 13]. The neurophysiological fact behind this idea is that we use the same parts and mechanisms of the brain for visual perception and for visual imagery. Therefore, whenever we simulate an event in our mind, we activate the same regions of the brain that we use in the physical world. Furthermore, Bergen points out that both kinds of images are indeed integrated. Visual imagery might interfere or enhance actual vision [BER 12, p. 28].
The other aspect that interests our study regarding mental images is the way in which they are understood as representation system. The earlier mentioned “connectionism approach” has found profound applications in computer science, particularly in digital imagery among other domains.
First of all, by the late 1930s, the basic components of computing were already invented (i.e. electrical circuitry, Boolean processing, propositional logics), and cybernetics established the umbrella field to explore relations soliciting the brain, the mind and the machine. As we can suspect, those different ingredients also constitute different levels of analysis and description. The question was: how to pass or move from one to another?
A simple way to understand a “description” is to think about a different language to describe something. What the computer central unit does can be described in machine language, but this is very difficult for humans because we do not think in binary terms. Thus, we go a step further to think in assembly language, which is easier and can be described in the decimal system. However, we know that, even today, assembly language is hardly taught in any “data visualization” workshop. What we do is go several steps higher. We describe in terms of another language what assembly does (and in consequence, what the machine language does at lower levels). We can now imagine the complexity of imbricated layers required for us to write a document in a word processor or to type a line of code in JavaScript.
