Google Brain Researchers Make Significant Progress On Language Modeling (arxiv.org) 32
New submitter integralclosure writes: Using neural networks, Google Brain researchers have significantly improved a computer's ability to model English (achieving extremely low perplexity score on a large dataset). Using the model they were able to generate random sentences, such as the following: 'Yuri Zhirkov was in attendance at the Stamford Bridge at the start of the second half but neither Drogba nor Malouda was able to push on through the Barcelona defence.' The sentences are generally coherent and mostly grammatically correct. Advances seem to be a replay of neural networks' dominance in the Imagenet competition.
Perplexity (Score:5, Funny)
"achieving extremely low perplexity score on a large dataset"
That's quite an achievement, since I have a very high perplexity score on that sentence.
Do they check whether it achieves low perplexity only when the sentence is not, in fact, perplexing?
Re: (Score:2)
if you read the article, you would know that the perplexity score is related to how perplexing something is, and the dataset is the number of examples of sentences.
Re: (Score:3)
I have the same issue with this one: Yuri Zhirkov was in attendance at the Stamford Bridge at the start of the second half but neither Drogba nor Malouda was able to push on through the Barcelona defence.
.
WTF? Sounds like chess.
Re: (Score:1)
Football, well soccer.
Stamford Bridge is the home ground of Chelsea. Drogba used to play for them. Dunno who Malouda was but Zhirkov was a Russian player who I don't think ever played for either Chelsea or Barcelona.
Re: (Score:2)
Oh, I though that was describing the technical breakthrough. ;-)
Made no sense at all.
Re: (Score:1)
Re: (Score:1)
After reading this, http://www.zonalmarking.net/2011/03/01/chelsea-2-1-man-utd-tactics/ [zonalmarking.net], the sentence "Yuri Zhirkov was in attendance at the Stamford Bridge at the start of the second half but neither Drogba nor Malouda was able to push on through the Barcelona defence." doesn't seem like much of an accomplishment. SubReddit Simulator [reddit.com] produces as much.
Re: (Score:2)
Re: (Score:2)
Not impressed (Score:3)
'Yuri Zhirkov was in attendance at the Stamford Bridge at the start of the second half but neither Drogba nor Malouda was able to push on through the Barcelona defence.'
Barcelona playing at Stamford Bridge? That hasn't happened for years!
Slashdot (Score:1)
They should hook this up to the Firehose and have it edit Slashdot stories.
Brought to you by the letters... (Score:2)
How well would it understand this? (Score:4, Interesting)
Time flies like an arrow. Fruit flies like a banana.
Re: (Score:2)
Time flies like an arrow. Fruit flies like a banana.
That's an ill-formed utterance though. Clever joke, but not really real English.
Re: (Score:2)
I think it's still a very useful thing to be able to understand though. English is absolutely loaded with ill-formed utterances in order to impart humor, sarcasm, intent, etc. It'll be very important for artificial systems to be able to reproduce and understand things like these.
Re: (Score:1)
Re: (Score:2)
Ummm... what exactly do you think is "not really real English" about it?
It's about what's called "generic reference". There are three generally used idioms in English for generic reference: 1) plural, zero article (fruit flies are insects); 2) singular, definite article (the fruit fly is an an insect) and 3) singular, indefinite article (a fruit fly is an insect). But that really only works for the subject of the subject of the sentence -- generic reference in grammatical objects is pretty much exclusively plural, particularly when dealing with the verb like, because we have th
Re: (Score:1)
Time flies like an arrow. Fruit flies like a banana.
That's an ill-formed utterance though. Clever joke, but not really real English.
Wrong. It's not a joke and it's perfectly legitimate English. Admitedly, it's an absurd statement, but that's the point.
Re: (Score:2)
Yes. Because it's not a simulacrum, it's a greatly simplified model. That it works at all suggests you've discovered at least a few of the important principles that let the vastly more complicated original work.
Also, you can poke at the model, see what improves it, what breaks it, plot receptive fields, all those things that are messy, difficult or unappreciated in an actual brain.
Fail (Score:2)
Yuri Zhirkov was in attendance at the Stamford Bridge
it's just "Stamford Bridge", not "the Stamford Bridge".
Try again!
Modeling Language (Score:1)
"The sentences are generally coherent and mostly grammatically correct," which means nobody will believe they are talking to a person.
I thought they said the program improved language modeling.