PRELIMINARY EXAM

QUESTION FOUR






By

Lynellen D. S. Perry







A Preliminary Exam Response
Submitted to the Faculty of
Mississippi State University
in Partial Fulfillment of the Requirements
for the Degree of Doctor of Philosophy
in Computer Science
in the Department of Computer Science






Mississippi State, Mississippi

February 199

QUESTION 4

In your dissertation topic, you describe the development of a neural network that learns the recurring 
patterns that exist in natural language.  Discuss how such a system might be used for natural language 
generation.  Your discussion should include a discussion of the following issues: (1) How would variety of 
expression be accomplished?  (2) What would the system need as input to the generation process?  
(3) How could the goals and state of knowledge of the audience be taken into account?  P.S. you may 
narrow this question so that you can give a more interesting answer.  For example, you may want to focus 
on one NLG task (e.g. answering database queries, generating poetry, etc.).  You may also want to only 
address one portion of the NLG process (deciding what to say, or determining high level text structure, or 
generating the text surface structure).  Please feel free to show how this approach would apply to the part 
of the process to which you feel it is most applicable.

	Natural language generation (NLG) consists of three stages: “(1) choosing the 
contents of an utterance; (2) preparing the plan of an entire text; and (3) the final 
realization of sentences according to the decisions taken in the two previous steps” 
(Mykowiecka, 1991).  Choosing the contents involves deciding which facts are relevant 
and adequate for the context, and deciding which facts are already known by the user so 
that new facts can be presented properly.  Preparing a plan involves creating a 
presentation order for the information, then deciding the subject, content, and form of 
each individual sentence.  Realization includes “ordering the sentence parts, choosing the 
proper words, and choosing the proper morphological forms” (Mykowiecka, 1991).  
There are many aspects to the realization of natural language.  For example, one 
NLG system considers the following elements of rhetorical style: formality, simplicity, 
timidity, partiality, detail, haste, force, floridity, color, personal preference, open-
mindedness, and respect (Hovy, 1990).  An NLG system must also consider discourse 
level aspects.  “Focus of attention constrains the information that needs to be considered 
when deciding what to say next.  It also provides constraints when the discourse strategy 
allows for several possible choices for what to say next by indicating which information 
ties in best with the preceding discourse” (McKeown, 1985).
There are several ways that neural networks might be incorporated into the NLG 
process.  Neural networks would not be best at storing the knowledge representations that 
are needed in the phrase where the NLG system decides on the contents of the utterance.
To be well-suited for generation, a knowledge representation framework must 
have (1) the ability to represent relationships between abstract and specific 
knowledge structures, (2) knowledge about relationships among concepts, 
including metaphorical relationships, used in constructing linguistic expressions, 
and (3) sufficient uniformity to allow linguistic and conceptual knowledge to 
interact in the selection and realization of linguistic structures (Jacobs, 1987).

A neural network is able to represent relationships between abstract and specific 
knowledge but only to a limited degree.  A neural network has a finite storage capacity so 
it could possibly be used in this way for a particular (small) domain but could not perform 
in a general sense.  As I have indicated in my response to another question, a neural 
network should be able to store knowledge about relationships between concepts, 
including metaphorical relationships.  Again, the capacity of the network is finite, so more 
traditional methods of knowledge representation would probably be best.
	Neural networks would be best used in the realization phase of NLG.  They have 
very little, if any, application to the text planning phase which follows the discourse 
planning phase.  The realization phase is also where one is concerned about the issues of 
expression variety and the goals and knowledge of the audience.  One example of neural 
networks be applied to the realization phase is found in Houghton’s paper as summarized 
in a collection by Dale et. al (1990).  Houghton’s work seeks to explore the “fundamental 
psychological, or neurophysiological, mechanisms which underlie the human capacity to 
produce sequentially structured behavior such as language” (Dale et. al, 1990).  So his 
neural networks are generating language at the word level by learning and recalling 
phonemic sequences, under certain constraints: “(1) Words are not stored literally, but are 
recreated on-line as dynamic activity patterns.  (2) There is no position-specific coding of 
elements.  (3) Upcoming phonemes in a word should be pre-activated, i.e., active before 
being produced (anticipation)” (Dale et. al, 1990).  Dale et. al also record Kitano’s 
connectionist work on simultaneous interpretation between English and Japanese in real 
time.
	Other ways that neural networks might be used in NLG include: creating variety of 
expression by modifying previously generated sentences to include appropriate metaphor; 
recognizing or classifying the knowledge level of the audience based on the vocabulary 
used in their interaction with the system; and predicting the next word in the sentence.
In the first case, the neural network would take as input the sentences that had 
been generated by the rest of the NLG system.  Content bearing words (versus function 
words) would be evaluated for the possibility of exchanging that word or its immediate 
phrase with a metaphor that was appropriately associated.  Perhaps this feature would 
only be enabled when the audience knowledge was sophisticated enough to properly use 
the abstraction that metaphor introduces.  For example, a very young person or a novice in 
the domain might not have enough specific knowledge to be able to understand the 
metaphor’s application to the particular situation.  On the other hand, I can also see 
appropriate metaphors helping the novice to transfer their knowledge from a familiar 
domain to the new domain in question.  The determination of this point would have to be 
a domain-specific part of the NLG system, I suspect, and not a part of the neural network 
which merely scans for appropriate contexts into which to insert a metaphor.
	In the second application mentioned above, the neural network would act as a 
classifier for an interactive NLG system.  It would help decide the level of knowledge 
possessed by the user, perhaps based upon a classification of the vocabulary used in 
queries or the topics concerned.  The network would be trained on the associations 
between vocabulary to knowledge level by presenting domain vocabulary and expecting 
the network to output an indication of the appropriate knowledge level.  Again, neural 
networks have a finite capacity for associating concepts, and associations that are valid in 
one domain may not be valid in others.
	Word prediction is reported in Elman (1995).  He says “language is a domain in 
which the ordering of elements is particularly complex.  Word order, for instance, reflects 
the interaction of multiple factors.  These include syntactic constraints, semantics and 
pragmatic goals, discourse considerations, and processing constraints” (Elman, 1995).  He 
shows how a recurrent neural network is used to predict the successive word based on the 
context thus far in the sentence.  So perhaps this type of network could be coupled with a 
NLG planning component that helped get the network started and kept it going in the 
right direction.
In conclusion, there are at least two instances in the literature where neural 
networks are being used to aid natural language generation.  I have also presented a few 
other possibilities for inclusion of neural networks into NLG system.  However, in any of 
these cases, the neural network would not be a stand-alone NLG system but would be a 
component (usually a post- or pre-processor) of a larger system.


REFERENCES
Dale, Robert, Chris Mellish, and Michael Zock. 1990. Current Research in Natural 
Language Generation. London: Academic Press.

Elman, Jeffrey L. 1995. Language as a dynamical system. Mind as Motion: Explorations in 
the Dynamics of Cognition, Robert F. Port and T. van Gelder, eds. Cambridge, MA: 
The MIT Press, 195-223.

Hovy, Eduard H. 1990. Pragmatics and natural language generation. Artificial 
Intelligence. 43:153-197.

Jacobs, Paul S. 1987. Knowledge-intensive natural language generation. Artificial 
Intelligence. 33:325-378.

McKeown, K. R. 1985. Discourse strategies for generating text. Artificial Intelligence. 
27:1-41.

Mykowiecka, Agnieszka. 1991. Natural-language generation – an overview. International 
Journal of Man-Machine Studies. 34(4): 497-511.