The Long Road from Text to Meaning

I stumbled upon this very interesting lecture with Adam Kilgarriff on google videos.

Key points from the talk:

• Approaches to language study: Rationalist vs. Empiricist


• Lemmatizers and Part-of-speech tagging


• Word sense, use and meaning


• Word sketching and thesaurus creation from corpora is discussed along with important problems such as representation and ambiguity.


• Using google as a NLP tool. Very interesting perspective!

Abstract: Computers have given us a new way of thinking about language. Given a large sample of language, or corpus, and computational tools to process it, we can approach language as physicists approach forces and chemists approach chemicals. This approach is noteworthy for missing out what, from a language-user's point of view, is important about a piece of language: its meaning.

I shall present this empiricist approach to the study of language and show how, as we develop accurate tools for lemmatisation, part-of-speech tagging and parsing, we move from the raw input -- a character stream -- to an analysis of that stream in increasingly rich terms: words, lemmas, grammatical structures, Fillmore-style frames. Each step on the journey builds on a large corpus accurately analysed at the previous levels. A distributional thesaurus provides generalisations about lexical behaviour which can then feed into an analysis at the ‘frames' level. The talk will be illustrated with work done within the ‘Sketch Engine' tool.

For much NLP and linguistic theory, meaning is a given. Thus formal semantics assumes meanings for words, in order to address questions of how they combine, and WSD (word sense disambiguation) typically takes a set of meanings (as found in a dictionary) as a starting point and sets itself the challenge of identifying which meaning applies. But, since the birth of philosophy, meaning has been problematic. In our approach meaning is an eventual output of the research programme, not an input.

Links

Adam Kilgarriff is a research scientist working at the intersection of computational linguistics, corpus linguistics, and dictionary-making. Following a PhD on "Polysemy" from Sussex University, he has worked at Longman Dictionaries, Oxford University Press, and the University of Brighton, and is now Director of two companies, Lexicography MasterClass (http://www.lexmasterclass.com) and Lexical Computing Ltd (http://www.sketchengine.co.uk/) which provide software, training and consultancy in the research areas.

Sketch Engine (SkE, also known as Word Sketch Engine) is a Corpus Query System incorporating word sketches, grammatical relations, and a distributional thesaurus. A word sketch is a one-page, automatic, corpus-derived summary of a word’s grammatical and collocational behaviour. You can try it using a free trial account.

Google sets creates a list of similar items given a few items. For instance, the set {apple,banana,strawberry} will result in a larger set with different fruits.

Human computation

Human computation is the concept of using humans to solve problems that computers suck at. This is the same sort of goal that artificial intelligence has; namely, solving problems that humans are good at but are difficult for computers.

Human computation might well be the next big paradigm of Artificial Intelligence, even though this sounds like a paradox. Efficient machine learning algorithms usually require supervised training to learn how to solve a given problem, but training can be a daunting task. In unsupervised learning, some sort of fitness or reward function is required. Writing a good fitness function can be really complex. For instance, it would definitely not be trivial writing a fitness function which determines if a picture contains pornographic material. Humans, on the other hand, have little trouble doing this. Karl Sims has suggested using humans as fitness functions in genetic algorithms, and applied the technique to evolving beautiful art.

I think that the potential effect of Human Computation in an AI context can be compared to the effect that web 2.0 has had on content on the web.

The difficult part is motivating people to collaborate in the training process.

Money for nothing and cycles for free

Human cycles are usually more expensive than computer cycles. That is, unless you get them for free. Luis von Ahn has found a clever way to get those cycles for free: By designing computer games where the players sort a specific kind of problems. The first batch of games he has designed deals with image classification. For instance the ESP Game is two player game where both players have to agree on a word for an image. The result is a set of classifications tags for the image.

The other games are Peekaboom and Phetch.




In this video Luis von Ahn explains the concept and gives some examples of "human computation" games he has designed.


I am especially excited about a not-yet-released game, Verbosity, which deals with the problem of creating a large corpus of common-sense knowledge. The MIT project ConceptNet has attempted to do this using web-collaboration. It's currently the best corpus around, but there is certainly room for improvement. Entering common-sense knowledge seems really boring, but this game might actually make the process fun enough to get people to collaborate.

Marvin Minsky interview

I just stumbled upon an interview with AI pioneer Marvin Minsky. The interview is mostly about his new book The emotion machine, "a machine that can switch between all the different kinds of thinking". I definitely want to read this book.

It's clear that Marvin Minsky has some profoundly different views on what AI should be like. He complains that the field is focused on solving problems with brain models and statistic models, but hardly no-one is working on making systems that can reason by analogy, or in other terms, think like a human.

He also talks about a bit about scifi, and if you are into that you will love this quote from the interview: "General fiction is pretty much about ways that people get into problems and screw their lives up. Science fiction is about everything else." I really laughed when I read it (because there is so much truth to it).

Colorless green ideas sleep furiously: Fun with a Ruby ChomskyBot

I recently fell over the concept of a Chomsky bot. A funny little thing which generates random paragraphs of text from a set sentence building blocks. It combines four kinds of phrases (introduction phrases, subject phrases, verb phrases and object phrases) into a sentence. The sentences this simple construction can create are amazing. They are syntactically correct and "hovers on the edge on understandability".

By the way, the title of this post "Colorless green ideas sleep furiously" is a syntactically correct but nonsensical sentence devised by Noam Chomsky. Noam Chomsky pioneered the field of generative grammars. The ChomskyBot implements a simple generative grammar.

The sentences generated by the bot are similar to the language of Noam Chomsky's works, and I guess the pun is intended.

Of course, I couldn't resist the temptation to write a Ruby version of the Chomsky bot:

class ChomskyBot 
  @@phrase_elems = [ "intro", "subject", "verb", "object" ] 

  def initialize(intro_file, subject_file, verb_file, object_file)
    @@phrase_elems.each do |e| 
      instance_eval("@#{e}s = []") 
      instance_eval("File.open(#{e}_file).each_line" + 
                    "{ |l| @#{e}s.push l.chop }")
    end
  end

  def generate_lines(n)
    lines = []
    n.times do
      @@phrase_elems.each do |e|
        eval("lines << @#{e}s.slice!(rand(@#{e}s.size-1)) << ' '")
      end
    end
   lines
  end

  def paragraph 
    generate_lines(5).join
  end
end

I tried to make it as simple as I could get away with. I shaved quite a few lines using eval, hope it doesn't hurt readability to much.

You'll need some phrase files to play with it. You can find those here:

Introduction phrases
Subject phrases
Verb phrases
Object sentences


You can try the original version of the ChomskyBot online. It's written in Perl (source) by Kevin McGovan. For more information, pay a visit to Chomsky bot inventor John Lawlers website.