How and why is a formal approach to multimodality fundamental to computer and transmedia literacy?

Abstract

Abstract: Central to a formal approach to multimodality is the definition of semiotic mode, which combines three interdependent levels: the material substrate, the technical characteristics organised into axes of description (structure) and the level of discourse semantics. The computational narrator maps formal regularities of an artefact/communicative situation, and demonstrates the formalisation of a multimodal discourse semantics in order to build new hypotheses about discourse coherence at different levels of detail. The work undertaken is situated both in terms of taking advantage of available technologies and developing ways of organising complex multimodal data. The methodological move is to develop abstract annotation layers which allow us to move on from the properties of measurable data to interpretations and characterisations in terms of narratives and strategies, under the challenge of getting to the meanings of patterns, rather than just finding patterns.