Научная статья

Информатика, кибернетика и программирование

The theory is necessary to analyze an existing state of the practice, to train new professionals and to use in the practical work. One may find two main approaches to a choice of foundations for the theories of the computer visualization and human-computer interaction which can be roughly described as psychological and semiotical. The paper contains discussion on the theories of computer visualization and human-computer interaction and considers the role of the theories in education and development processes.



72 KB

0 чел.


Vladimir L. Averbukh

Institute of Mathematics and Mechanics, Ural Branch of the Russian Academy of Sciences

16, S.Kovalevskaja street, 620219, Ekaterinburg, RUSSIA


The theory is necessary to analyze an existing state of the practice, to train new professionals and to use in the practical work. One may find two main approaches to a choice of foundations for the theories of the computer visualization and human-computer interaction which can be roughly described as psychological and semiotical. The paper contains discussion on the theories of computer visualization and human-computer interaction and considers the role of the theories in education and development processes.

Key words: Computer Visualization, Human-Computer Interaction, Psychology, Semiotics


The discipline of Human-Computer Interaction has been developed over the past fifty years since significant number of the computers work stably. Now a considerable part of all world programmers work in HCI domain. If in the first decade of HCI history the number of active users was near thousands now the possibility of human-computer interaction are available to some extent to almost all population of Earth.

At the same time, the situation of human-computer interaction and visualization is ambiguous. In some cases, the computerization of management processes leads to increasing of queues and problems in institution activities. Some of interfaces applied in mass services are inconvenient. They force people to spend considerable effort to learn and to use, and even lead to serious stress in some cases. The problems are related, in our opinion, the inadequate regard of what is traditionally related to human factors. Below, referring to the design of human-computer interaction and computer visualization we’ll mean design of “the human factor” elements for visual interactive systems.

The analysis based on the theory (or more exact theories) of human-computer interaction and computer visualization must develop to overcome the problems. And these theories are under developing. A lot of conferences devoted to visualization and HCI hold yearly. In many universities around the world special departments and research and educational institutes on Human-Computer Interaction and Computer Visualization have established. Note, however, that some practitioners do not know anything about the results of the research. Other practitioners expect from the theory only councils on design of the interface which is reduced to a choice of colors or placement of visual objects on the screen. Sometimes they said that there is no theory in human-computer interfaces and visualization, or that such theories are not necessary, in principle, because everything works fine without any theories.

On our opinion the theory is necessary, firstly, to analyze an existing state of the practice, secondly, to train new professionals and thirdly, (and this is the main) to use in the practical work. Without the theory there are no reliable methods of adopting and sharing really valuable experience, instead of the casual ideas which have appeared in connection with a certain level of hardware for interfaces and Computer Graphics and/or Software Engineering.

The scientific theory should satisfy some requirements. (See also (Workshop at Visweek, 2010).) Among them there are the discipline structurization, supporting of analytical functions in its frameworks, and the prediction of new phenomena. One may say about the explanatory and predictive force of the theory. Thus, on the basis of the satisfactory theory (at a given period of the discipline development) one may analyze and explain any known phenomena, predict the emergencies of new phenomena, concepts and facts, carry out a systematic description of the discipline as a whole. Thereby there is a possibility to fix available achievements, to transfer them in courses of study, to create conditions for the further development of the discipline. An important result of the satisfactory theories of HCI and Computer Visualization should be a scientific basis for the quality design, the development and the evaluation of HCI and visualization systems. Below we’ll consider approaches to the theories of human-computer interaction and visualization which in our opinion form a base of designing, developing and education in this area.

2. “Activity” approach to theory of human-computer interaction

Dealing with interfaces it is naturally to pay attention to psychological Activity theory developed in Soviet Union in the middle of the XX century. Activity theory is connected, first of all, with names of A. Leontev and S. Rubinshtein. (Leontev, 1978 and Rubinshtein, 2005) The “activity” approach, actively developed since the late 80s, is one of the main approaches to the theory of human-computer interaction. The set of important papers on this subject was published beginning from the 80th and 90th years (for example Kaptelinin, V., 1996; Nardi, B., 1996; Kaptelinin, V. 2012; Rogers, Y., 2004). Just recently the monograph by Victor Kaptelinin and Bonnie A. Nardi was published (Kaptelinin, V., & Nardi, B., 2012.).

When acquainting with Activity theory it seems that the theory was specially created in 30-s' year of the XX-th century for the future human-computer interactions so exactly its ideas are fit on the specifics of HCI. Activity theory can be supported by results of Psychophysiology theory developed about the same time. So this psychophysiologal “reinforcement” forms psychological and physiological basis of the theory of HCI. The Activity is considered as the conscious and purposeful process. The analysis of activity has to forego interface design. This analysis includes revealing of the activity purposes and ways of this purpose achievement, a determination of an understanding level for this purpose by a person and a definition of activity motives.

During the activity an action is realized, when the partial result, which it is reached, becomes a direct objective of the subject, and ceases to be realized when the target moved on and the former effect becomes only way to carry out other actions directed at a common goal. The action directed on the small purposes, is switched off from consciousness, passes in the subconscious. (Rubenstein, 2005)

One of the basic principles of activity theory is the hierarchical structure of activity. Activity theory differentiates between processes at various levels (or, rather, groups of levels), taking into consideration the objects to which these processes are oriented. Activities are oriented to motives, that is, the objects that are impelling by themselves. Each motive is an object, material or ideal, that satisfies a need. Actions are the processes functionally subordinated to activities; they are directed at specific conscious goals. According to activity theory, the dissociation between objects that motivate human activity and the goals to which this activity is immediately directed is of fundamental significance. Actions are realized through operations that are determined by the actual conditions of activity. (Cited by Kaptelinin V., 1996) Thus, the hierarchy is established: activity – the realized actions – operations. The activity is divided into a set of conscious and motivated actions which, in turn, implemented a set of operations.

All activities of the system and its various changes one may present entirely in terms of results, which further emphasizes its critical role in the behavior of the system. This activity can be fully expressed in the questions, reflecting different stages of the system:

  •  what the result should be obtained?
  •  when the result should be obtained?
  •  whereby (what mechanisms) the result should be obtained?
  •  how the system verifies the adequacy of the obtained result? (Anokhin P., 1978)

Passivity of the dominant excitation, i.e. the dominant of the moment, can be considered as a source of passivity of perception and, consequently, a false perception. At the same time, this passivity provides stability of attention and a structurization of the received information. (Ukhtomsky A., 2002) Activity to be effective and adequate, should always be corrected and be free all time of predefined schemas. (Bernstein N., 1947) However, the goals in the framework of an interface should not be achieved through a complex action, becoming an autonomous activity. The task of the designer is to minimize the complexity of the activity within the interface and to provide systemacity of interfaces. One may consider systemacity of interfaces as using similar actions to achieve the similar goals in different interface actions. Design of activity, actions and operations requires knowledges on the areas which can be attributed to Psychophysiology. Also it is important to take into account psychomotor factors of interface actions. Simplicity and ergonomics are especially important for mass interfaces.

The activity approach to designing human-computer interaction involves the analysis of the problem and description of action to tackle this problem. These activities consist of sets of conscious, motivated by goal achievement actions. In turn these actions are reduced to sets of operations. At each level of the hierarchy, a designer needs to reveal and to define clearly objectives, which are related to the implementation of activities. The design of operations needs to pay attention to the principles of systematicity. The development of mass and professional interfaces requires solutions of a number of additional problems.

Let's consider “instrumental” interfaces in connection with “Activity” approach.

One may understand instrumental professional interfaces as interfaces for practioners in the areas where a substantial part of the activities is interaction with people. That is, we consider the interfaces for various categories of clerks, health workers, salesmen, using them to perform their functions, as a tool their professional activities.

The mass instrumental type of interfaces includes “nonprofessional” general-purpose interfaces, using, for example, for tickets reservation systems, for health, banking, social and public services, etc.

“Recreational” interfaces (that is interfaces using in social networking nets, communicating services, games, and so on) are outside of our attention. Also we don’t consider interfaces for professionals dealing within information technologies. To a marked degree this limitation is connected with quality and usability criteria applied for these interfaces, such as time spent on the page, the number of “clicks” on some image, subjective evaluation of a small number of respondents, etc. These criteria are not suitable for “instrumental” interfaces. The quality criteria in this case should be based on evaluation of user activity results.

In the case of mass instrumental interfaces quality can be measured, taking into account the time spent by users to obtain the result and the level of stress during the result was reached. In this regard, laconic interfaces with the minimal requirements to user memory and attention are necessary. Hence we need to save and restore the current interface state and context. Interfaces that use the principles of the menu or any of the programming techniques are unsuitable here.

For professional instrumental interfaces the quality criterion may be measured through the number of people satisfied with the work of the institutions during some period of time. That is, we consider the number of customers, patients, clients, etc. who reached result satisfying them and not received serious stress. Thus, the level of stress of professionals using the interface is measured indirectly.

Stress may be measured both by interviews of interface users or visitors of the institution and by laboratory and “field” studies of their physiological indicators that reflect the level of stress.

The size of time period depends on the term demanded for the decision of the given problems, for example, in any cases it is a working day, in any – a week, a month or even a year.

In case of “mass” interfaces the designer, formulating requirements to the interface, participates in formation of the future user activity. The user can't refuse to use the system because by using the interface she/he gets access to vital services, resources, information, etc. “Mass” interfaces have be intended to a “weak unit”. That is a person with minimal capacity to perceive and analyze information has to use this interface successfully.

In the case of “professional” interfaces the goal of user activity is predetermined beforehand. The problem under decision dictates the requirements for the interface generally. A “professional” also (as a user of “mass” interfaces) can't refuse to use the interface because her/his activity is strictly regulated.

The designer of the interface should study the goals and features of the given activity. She/he must not deform the activity and/or bring additional complexities to it. In our opinion it should be refused to use any intricate tuning in “professional” interfaces. Altogether it should avoid to use anything that may be considered as programing because programing is another activity supplementary to the main duties of a “professional”.

Within frameworks of the activity a “professional” (that is a clerk, a health worker, a salesman) deals with a certain set of entities. For example, she/he handles personal documents, fills the form of internal documents, interacts with visitors, and sometimes takes the money and makes bills and receipts. Computerisation adds a new type of activity and creates a new entity - the human-computer interaction. One can watch examples of interfaces that continuously switch the clerk attention, prevent to interact with visitors, overloading her/him by additional tasks.

It is necessary to analyze the activity generated by the “instrumental” interfaces from the standpoint of a possible “redundant” and “insufficient” level of computerization. In general it is necessary to reduce (instead of to increase) the number of entities that a “professional” has to process. That is why the designed interface should assume completely the functions of processing one or another entity. Then the interface will not be a new, additional and complicating entity in professional dealing.

Designing “instrumental” interfaces is inseparable from generic issues in the institution where interfaces will be applied such as the correct organization of operations and document processing, data confidentiality and so on. These solutions tend to lie outside the competence of designers. But without them, all efforts can go down the drain.

3. The bases of the computer visualization theory

Other important task is to format the computer visualization theory.

Officially, as an independent discipline Computer Visualization traces its history to 1987 by Special Issue of ACM SIGRAPH Computer Graphics “Visualization in Scientific Computing”. (Visualization in Scientific Computing, 1987) The description Computer Visualization as the independent discipline summed up the great practice of Computer Graphics since beginning of 60-th. In this issue the main conceptions of the new discipline were defined. The visualization is considered as a method of computing. It transforms the symbolic into the geometric, enabling researchers to observe their simulations and computations. Visualization offers a method for seeing the unseen. The goal of visualization is to support the analysis and interpretation stages in framework of the computer modeling cycle.

One of the most popular approaches to a choice of foundations for the theory of the computer visualization is based on the theory of perception of the visual information. This approach is considered in in a set of publications (for example, (Workshop at Visweek, 2010)). Note in this connection the publication of the research group B. Tversky focusing on the perception of how individual elements graphic display (color, shape, texture, etc.), and integrated graphical output (including animation). (Zacks, J., Tversky, B 1999; Tversky B. 2001; Tversky B., Morrisony J.B., Betrancourt M., 2002; Tversky B. 2005). The example of the interesting researches on this topic see also in (Ziemkiewicz Caroline, Kosara Robert, 2010).

Gestalt theory is widely used in the design of the images on a computer screen for human-computer interaction and computer visualization systems (for example Graham L., 2008; Fraher R., Boyd-Brent J., 2010 and Soegaard M., 2011).

Thus, we have the theoretical bases for the correct designing of visualization from positions of the image perception.

However, the perception is only the first stage of the interpretation of visual images. Exactly interpretation is the main task of computer visualization in the computer modeling cycle. The recearch of interpretation is traditionally held in the framework of semiotics.

4. A “semiotics” approach to the theory of visualization and human-computer interaction

A “semiotics” approach to the theory of visualization and human-computer interaction began to develop in the 80-th years of the twentieth century. The statements of the classical semiotics were used to describe visual sign processes in connection with a computer graphics and visualization. Our researches on problems of the theory of computer visualization and computer metaphors are based on the semiotics approach. It is shown that the human-computer interaction and visualization have a semiotic nature. The conceptions of a visualization language and a figurative (visual) text described on this language are considered. The computer metaphor is considered, as a basis of the visualization language. The semiotics analysis of computer metaphors allows to evaluate known metaphors and to search new ones for specialized visual systems (Averbukh V.L., 2001; Averbukh V.L., 2005; Averbukh V. et al., 2007; Averbukh V. et al., 2008).

Thus, the semiotics analysis can be an important tool for the visualization systems design and development.

Semiotics, dealing with sign systems and with practice of their functioning, may be considered as tools for descriptions of theories of HCI and Computer Visualization just as Mathematics is tools for descriptions of Physics Theories.

The obvious semiotic nature of the human-computer interface and visualization allows to reveal sign systems that determine interactions, visualization and communications. Human-computer interaction in this connection may be described precisely as sign process. Visualization also may be described as sign process similarly to human-computer interaction. Processes of human computer interaction and visualization contain user interpretation of visual and dialog objects as their essential part. In turn the process of sign interpretation is researched in frameworks of semiotics. That is why one may consider semiotics as the base of theories of HCI and Computer Visualization.

If human-computer interface and visualization have the sign and language nature then each interface and visualization system contains the language as its core. The language in this case is understood as the systematical description of entities under consideration, methods of their representation, modes of changes of visual display, as well as, techniques of manipulations and interaction with them. The language (or rather a base sign system) is built upon some basic idea of similarities between application domain entities with visual and dialog objects, i.e., upon a computer metaphor (that is interface metaphor and visualization metaphor).

We consider the conception of “metaphor action” that is important for the analysis of computer metaphor. This conception has formed a basis for the analysis actions of concrete interface and visualization metaphors. The analysis has to reveal criteria for evaluation of metaphors and for its searching and selecting. Computer metaphors promote the best understanding of interaction and/or visualization semantics, as well as provide visual representation of the appropriate objects and determine the user's manipulations set. A metaphor, considered as a basis of the sign system, underlies in a basis of an interactive visualization language in its turn.

The understanding of a metaphor as a sign system gives us a basis for evaluations of metaphors offered in concrete cases. If the used affinity (comparison or a set of comparisons) matches the systemness requirements, then we may speak about existence of a useful metaphor. “Semiotic” approach to HCI and computer visualization theories makes it possible to choose computer metaphors as the key point of HCI and computer visualization systems design and development. Analysis of metaphors is the useful tool for this design. Also the analysis forms the set of criteria for evaluation of metaphors. One can choose a metaphor, as well as construct on its base a correct set of views for a visual interactive system. Criteria of a choice may be considered as criteria of metaphor quality.

5. Conclusion

Thus, one may find two main approaches to a choice of foundations for the theory of the human-computer interaction and computer visualization which can be roughly described as psychological and semiotics. On this basis one may costuct theoretical foundations of ergonomics for interfaces and visualization design. Also it is necessary to study the principles of design as an integral part of the study of those aspects of psychology, which should be used in the developmant of human-computer interaction and visualization systems. Principles of Art Design are largely based on the principles of perception (but were not limited to). Interface Design principles are based on the universal principles of Design and Visual Arts, but are not limited by them. As usually in Interface Design only statical conditions are considered. Even the dynamics of the transition from one view to another are understood poorly. All the more so, there are no recommendations about creation of dynamic visualization. In this plan the experience of the theory of cinematograph is important. One may use ideas of theory of cinematograph to design and develop human-comuter interfaces and visualization. The semiotics approach to the cinematograph theory one may pick up from books and articles of Sergey Eisenstein, Victor Shklovskiy, Yurii Lotman. (Eisenstein S. 1949; Shklovskiy V., 1985; Lotman Yu., Tsivyan Yu., 1994). Understanding and analysis of film and animation meaning may be studied by the experience of Walt Disney.

Here are some of the work directions for professionals in psychology, semiotics, together with designers of interfaces. In addition, the adequate mathematical formalization is necessary for normal development of the theory. However, many are not clear. This work is far from complete. However, many are not clear. This work is far from complete.

In conclusion, we emphasize again that the theory is not needed for the theory, but we need the theory as a basis for designing proper human-computer interaction and visualization. . Our ideal is to develop personalized computer systems that are configured for a specific user, considering features of her/his perception and thinking.

The meeting place of theory and practice is Academia, where it is natural to develop and study the theory for professional training of future developers.


1. Anokhin P. (1968) Biology and Neurophysiology of Conditioned Reflex. M. (in Russian)

2. Averbukh V.L. (2001) Visualization Metaphors. Programming and Computer Software. Vol. 27, No. 5, pp. 227-237.

3. Averbukh V.L. (2005) Toward Theory of Computer Visualization Computational Technologies V.10, N 4, pp. 21-51. (In Russian.)

4. Averbukh V., Bakhterev M., Baydalin A., Ismagilov D., Trushenkova P., Interface and Visualization Metaphors (2007) J. Jacko (Ed.): Human-Computer Interaction, Part II, HCII 2007, LNCS 4551, Springer-Verlag Berlin Heidelberg, pp. 13-22.

5. Averbukh V.L., Bakhterev M.O., Baydalin A.Yu., Gorbashevskiy D. Yu., Ismagilov D.R., Kazantsev A.Yu., Nebogatikova P.V., Popova A.V., Vasev P.A. (2008) Searching and Analysis of Interface and Visualization Metaphors  Human-Computer Interaction, New Developments. / Edited by Kikuo Asai. Chapter 3, Vienna, In-teh. ISBN 978-953-7619-14-5, pp. 49-84.

6. Bernstein N. (1947) Motion Synthesis M. Medgiz. (in Russian)

7. Eisenstein, Sergei (1949), Film Form: Essays in Film Theory, New York: Hartcourt; translated by Jay Leyda

8. Fraher R., Boyd-Brent J. (2010) Gestalt Theory, Engagement and Interaction // Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems ACM. pp. 3211-3216.

9. Graham L. (2008) Gestalt Theory in Interactive Media Design // Journal of the humanities and social sciences, 2(1), pp. 1-12.

10. Kaptelinin, V. (1996) Activity Theory: Implications for human-computer interaction. In B. Nardi, (ed), Context and Consciousness: Activity theory and human-computer interaction. Cambridge, MA: MIT Press. P. 107 – 110.

11. Kaptelinin V. (2012) Activity Theory // Encyclopedia of Human-Computer Interaction. Chapter 16.

12. Kaptelinin, V., Nardi, B. (2012.) Activity Theory in HCI: Fundamentals and Reflections. Synthesis Lectures on Human-Centered Informatics. Morgan & Claypool. 2012.

13. Leont'ev, A. (1978). Activity, Consciousness, and Personality. Englewood Cliffs, NJ: Prentice-Hall.

14. Lotman, Y. Tsivyan Y. (1994) Dialogue with the Screen. Tallinn, Alexandra. (in Russian)

15. Nardi, B. (1996) Activity Theory and Human-Computer Interaction In B. Nardi, (ed), Context and Consciousness: Activity theory and human-computer interaction. Cambridge, MA: MIT Press. P. 7-16.

16. Rogers, Y. (2004) New Theoretical approaches for Human-Computer Interaction. Annual Review of Information, Science and Technology. 38. Pp. 87-143.

17. Rubinshtein S. (2005) Principles of General Psychology. Moscow. Piter. (in Russian)

18. Shklovskiy V. (1985) The Semantics of a Film In 60 years working in cinema. Iskusstvo, pp. 30-32. (in Russian)

19. Soegaard M. (2011) Gestalt principles of form perception

. Tversky B. (2001) Spatial Schemas in Depictions // Spatial Schemas and Abstract Thought, MIT Press, Cambridge, MA

21. Tversky B. (2005) Prolegomenon to Scientific Visualizations // Visualization in Science Education. Models and Modeling in Science Education, Springer, Volume 1, Section A, pp. 29-42.

22. Tversky B., Morrisony J.B., Betrancourt M. (2002) Animation - can it facilitate // Int. J. Human-Computer Studies. 57, pp. 247-262.

23. Ukhtomsky A. (2002) Dominance. St. Petersburg, Piter, (in Russian)

24. Visualization in Scientific Computing (1987) Special Issue, ACM SIGRAPH Computer Graphics, V. 21, N 6, November.

25. Workshop at Visweek (2010) The Role of Theory in Information Visualization, October 25, 2010, Salt Lake City, Utah, USA


26. Zacks, J., Tversky, B (1999) Bars and lines: A study of graphic communication // Memory and Cognition, (27), pp. 1073-1079.

27. Ziemkiewicz Caroline, Kosara Robert (2010) Implied Dynamics in Information Visualization, Proceedings Advanced Visual Interfaces (AVI), pp. 215-222. 


А также другие работы, которые могут Вас заинтересовать

2459. Відокремлені прикладки, урок для 8 класу 76.5 KB
  Мета організації уроку: сформувати в учнів поняття про відокремлену прикладку на основі відтворення і поглиблення знань, умінь і навичок учнів з тем, отриманих на попередніх заняттях, навчити восьмикласників обґрунтовано вживати розділові знаки при відокремлених прикладках, знаходити і виправляти пунктуаційні помилки на вивченні правила.
2460. Особливості побудови наукового і художнього описів. Опис тварини. Усний вибірковий переказ тексту, що містить опис тварини 35.72 KB
  Мета організації уроку. Сформувати в учнів поняття про особливості побудови художнього і наукового описів тварини, навчити правильно будувати текст-опис тваринки, знаходити основне, неповторне в образі тваринки і описувати свої спостереження, навчити школярів сприйняття та відтворення текстів, що містять опис тварини.
2462. Присвійні, вказівні займенники, їх відмінювання 74 KB
  Мета організації уроку. Сформувати в учнів поняття присвійних та вказівних займенників, уміння їх відмінювати.
2463. Відокремлені додатки 59.5 KB
  Мета організації уроку. Сформувати поняття про відокремлений додаток, навчити оформляти відокремлені додатки на письмі та використовувати набуті уміння на практиці.
2464. Степан Васильченко. Свекор 54 KB
  Ознайомити учнів з фактами життя письменника, які вплинули на формування світогляду, специфікою його майстерності та із змістом оповідання Свекор, розвивати навички виразного читання, переказу прозових творів; виховувати почуття любові, злагоди, взаємодопомоги та турботи у сім’ї.
2465. Письмовий твір-роздум на морально-етичну чи суспільну теми публіцистичного стилю 39 KB
  Мета організації уроку. Удосконалювати вміння учнів продукувати письмовий твір-роздум на морально-етичну тему, розвивати усне і писемне мовлення, сформувати в учнів вміння виділяти в тексті всі компоненти роздуму, розуміти їх призначення в організації висловлювання, а потім свідомо відтворювати їх у власному мовленні.
2466. Усний твір, що містить опис тварини. 33.38 KB
  Мета організації уроку. Сформувати в учнів поняття про опис зовнішності тварини; навчити правильно будувати текст-опис зовнішності тварини, через зовнішність описати її характер, знаходити основне, неповторне в образі конкретної тварини і описувати свої спостереження.
2467. Загальновживані (нейтральні) та стилістично забарвлені слова. 33.86 KB
  Мета організації уроку. Сформувати в учнів поняття про загальновживані та стилістично забарвлені слова, навчити розпізнавати і використовувати їх у власному мовленні, виховувати любов до слова.