How do we go about producing a work if not from a conventional beginning?
It may start with a germ of an idea, this text. Maybe it’s a clearly defined vision. Or it may start where it ends, with the creator working backwards. Or it could be nothing more than a random string of disconnected words, fragments that must be pieced together like a quilt, layer upon layer.
It has to come from somewhere.
The point is that if words and images have histories, we are already pulling from the available designs, as the New London Group does (Wysocki, p. 26). These conventions shape our thoughts before we even realize we have them. The framework we are most familiar with, that of words in narrative form, is a comfortable convention, and usually our starting point for creation. Even if we are rebelling against the format, it is still the starting block.
In this sense, we can see our own contributions as the latest in a series of utterances, which are always connected and reflective, what Russian theorist Mikhail Bakhtin called intertextuality. From Bakhtin:
“Any concrete utterance is a link in the chain of speech communication of a particular sphere. The very boundaries of the utterance are determined by a change of speech subjects. Utterances are not indifferent to one another, and are not self-sufficient; they are aware of and mutually reflect one another.” (1986, p. 91)
While Bakhtin primarily meant utterances in the shape of speech acts, his idea of genre is encompassing and includes texts of all kinds. I will use the word text as defined by Kristin Arola, et al, to mean the piece of communication as a whole – the visuals, sounds, and movement that make up multimedia.
We also tend to see the world as dichotomous, as Wysocki points out, referencing Marshall McLuhan, W.J.T. Mitchell, and Aristotle.
Those opposites include words/images, internal/external, mind/body, masculine/feminine, time/space, rest/motion (p. 27). There are reasons for this, namely to do with order and classification, which help us make sense of the natural world.
But it is possible to shape texts dialogically and not just dialectically, which is how I prefer to approach multimodality. Bakhtin also gives us that theoretical lens to look through. In a dialogic context, then, ideas are forever being shaped and reshaped.
Take, for instance, Alison Bechdel’s graphic novel Fun Home. Here she provides obvious contrasts between her and her father’s experiences, but she also gives us several literary comparisons to create a dialogical conversation. There is no one true answer to get at, as there would be with a dialectic process. There is only the interpretation of these texts through Bechdel’s reconstruction of her life.
Then how do we find identity?
If we continue with Bakhtin, we see the meaning in our texts isn’t exclusively our own, that language “is populated – overpopulated – with the intentions of others” (1981, 294).
What does all of this say about our identity, then? Are we then part of a collective, just creating another iteration of a story and slapping the label “new media” on it? Well, yes.
But it is how we then take the text and imbue it our intention that gives rise to new meaning.
Of course, it is our stamp of individuality that makes our text distinct from others. Our deliberate choice of modes (Arola 4) affords us varying degrees of personalization, depending on the technology and genre. If we choose the Vine app for content, for instance, we are selecting to create a video, which covers a variety of modes (if not all). We are also open to be as creative as a video format allows, aware of the affordance that gives us the chance to communicate emotion in an immediate way (Arola 15).
“We see ourselves in what we produce,” Wysocki asserts (25). That act of production, which involves numerous rhetorical decisions, reveals our identity. This retrospective self-awareness is an intriguing prospect, but not without its problems.
If what we create is another link in the long chain of utterances, then how can we truly ever escape the past and shake off a history that frames the world in a series of dichotomies? This is the question Wysocki has, too, and one that I think is being answered by artists and students and creators every day.
What makes a text multimodal?
In Multimodal Discourse, Gunther Kress and Theo van Leeuwen define multimodal design as “a deliberateness about choosing the modes for representation, and the framing for that representation” (45).
They see the multimodal model changing as technology – and our grasp of it – evolves. No longer are we confined to a framework where a single mode does a single task. Now we not only have the freedom to combine modalities, we can determine what each mode should signify at any given point.
Kress and van Leeuwen, again:
“Of course there were other modes of representation, though they were usually seen as ancillary to the central mode of communication and also dealt with in a monomodal fashion. Music was the domain of the composer; photography was the domain of the photographer, etc. Even though a multiplicity of modes of representation were recognised, in each instance representation was treated as monomodal: discrete, bounded, autonomous, with its own practices, traditions, professions, habits. By contrast, in an age where the multiplicity of semiotic resources is in focus, where multimodality is moving into the centre of practical communicative action.”
I’d like to explore the multimodality of comics and graphic novels to show that there is complexity of composition even in the most unlikely places, such as the seriocomic Hyperbole and a Half by Allie Brosh, who makes use of crude, childlike illustrations to convey her often-dark subject matter.
The discordance is striking – is this highbrow or lowbrow? Does it even matter? This is a theme Wysocki addresses throughout her essay. History tells us words are superior. Images, decidedly less so. Comics held no intellectual stimulation or challenge.
And if we continue to see words and images as “standing in formal opposition,” then other contrasts, as those mentioned earlier, begin to crop up as well.
For instance, the central figure in Hyperbole, drawn in such a way that is genderless, leaves one questioning the masculine/feminine. Or maybe, as I related to it, removes the question altogether.
The comic, somber yet hilarious in its depiction of depression, is disconcerting on many levels. We shouldn’t make light of the situation described here, yet we see the absurdity in the image, and the futility of depression becomes all the more clear.
This in-betweenness, the idea that this text defies our standard notion of what should be, is precisely what makes Brosh’s text so effective (37). The words and images aren’t acting in opposition to each other – they synthesize to create a narrative. Take away one, and meaning is lost.
From the National Council of Teachers of English:
“All modes of communication are codependent. Each affects the nature of the content of the other and the overall rhetorical impact of the communication event itself.”
By playing on the strengths of certain modes at certain times, Brosh is making use of affordances. As pointed out in Writer/designer, the visual mode is a way to communicate emotion in an immediate way, while the linguistic mode affords us the time we need to digest a passage (Arola 15).
We are to the point now where, as N. Katherine Hayles asserts, “we think through, with, and alongside media” (1).
Her concept of contemporary technogenesis, or the idea that humans and technics have coevolved together and continue to shape one another, is providing us new ways of creating using new, and what I would consider neutral, dichotomies (81). These include the narrative (temporal) and database (spatial).
Arola et al also hint at this technogenesis in their case study of the US government economic recovery maps. The symbiotic relationship between the data and narrative evolves as the technology does, providing a more effective text.
Ideas for further investigation
If we are to create in digital media, how do we know we are crossing into uncharted territory? What makes this different from the old?
What does a digital narrative look like? I found this quote by Ted Nelson, from Frank Chimero’s discussion of screens, sums up my feelings nicely:
How can we improve on multimodal? Expand on it? Shed the past and its conventions?
Are we aware of our influences and how they shape our choices?
Do we know before we begin how a particular media will be layered with another?
What is the act of creating, then?
Sources:
Arola, Kristin L., Jennifer Sheppard, and Cheryl E. Ball. “What Are Multimodal Projects?” Writer/designer: A Guide to Making Multimodal Projects. Bedford/St. Martin’s, pag. Print.
Bakhtin, M. M. The Dialogic Imagination: Four Essays. Ed. Michael Holquist. Austin: U of Texas, 1981. Print.
Bakhtin, M. M. Speech Genres and Other Late Essays. Ed. Michael Holquist and Caryl Emerson. Trans. Vern McGee. Austin: U of Texas, 1986. Print.
Hayles, Katherine. How We Think: Digital Media and Contemporary Technogenesis. Chicago: U of Chicago, 2012. Print.
Kress, Gunther R., and Theo Van Leeuwen. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold, 2001. Print.
Wysocki, Anne Frances. “Drawn Together.” Composing(media) = Composing(embodiment): Bodies, Technologies, Writing, the Teaching of Writing. Logan, UT: Utah State UP, 2012. N. pag. Print.
One, Wild & Precious 