Baidu Forced To Withdraw Last Month's ImageNet Test Results 94
elwinc writes: Back in mid-May, Baidu, a computer research and services organization in Mainland China, announced impressive results on the ImageNet "Large Scale Visual Recognition Challenge," besting results posted by Google and Microsoft.
Turns out, Baidu gamed the system, creating 30 accounts and running far more than the 2 tests per week allowed in the contest.
Having been caught cheating, Baidu has been banned for a year from the challenge. I believe all competitors are using variations on the convolutional neural network, AKA deep network. Running the test dozens of times per week might allow a competitor to pre-tune parameters for the particular problem, thus producing results that might not generalize to other problems. All of which makes it quite ironic that a Baidu scientist crowed "Our company is now leading the race in computer intelligence!"
Lead from behind.... (Score:4, Insightful)
That's what I always say.... (/sarcasm)
WHAT! (Score:5, Insightful)
Chinese company caught cheating? NO WAY!
Seriously though, raise your hand if you're surprised.
Re:WHAT! (Score:5, Funny)
it's not called cheating there. the ideograms translate roughly to "auspicious cooperation with jade dragon of opportunity."
Re:WHAT! (Score:5, Funny)
I think that's from Confucius's 65th Rule of Acquisition.
Re: (Score:3)
I think you're mixing the Chinese with the Ferengi here...
Re: (Score:3)
Congratulations, you killed that joke dead.
Re: (Score:2)
Hurray!
Re:WHAT! (Score:4, Interesting)
it's not called cheating there.
Except it wasn't "there". It was "here". Baidu's main research lab, where this AI research was done, is in Sunnyvale, California, and many employees there are American citizens.
Re: (Score:3)
Corporate culture is corporate culture. Location is largely irrelevant.
Re: (Score:2)
the arXiv paper presenting the results was authored by five people all with Chinese names. at least one of them is fresh from Beijing; i didn't bother tracking down the rest.
i'm sure a lot of research goes on there, by all sorts of people. Andrew Ng seems like a good guy. none of this is directly relevant to the submissions. (though neither is any of this that important, really.)
Re:WHAT! (Score:4, Funny)
Chinese company caught cheating? NO WAY!
I almost broke my string of pearls from clutching them so hard. You just don't expect this kind of corruption out of China.
Re: (Score:3)
I just got the vapors.
Re:WHAT! (Score:5, Insightful)
Chinese company caught cheating? NO WAY!
Seriously though, raise your hand if you're surprised.
I'm not surprised, but not for the reason I suspect you have in mind: "Because Chinese are just so and so...".
However, it does not really surprise me that we see this from some Chinese companies. China is a developing nation, and they are still relatively new to the way companies play it in the West - not that we are in fact more honest in the West, we have just learned how and when to be dishonest in a way that doesn't make as much noise. I mean, just think of large corporations that avoid paying tax or buy cheaply from sweat-shops employing child-labour. There is no shortage of examples.
But there is another thing in it: lack of regulation. It should not be a surprise to anyone that when there is too little regulation, the most ruthless will feel entitled to bully others - the free, unregulated market can never work to the benefit of everybody, because there will always some, that ruthlessly go for maximising their own short-term advantage, and and that behaviour pushes out the competition and creates monopolies. It is perhaps ironical that this argument is exactly analogous with the argument against Communism: "people are selfish, so if they don't have a reason to work harder, most won't".
We see this in all developing countries, but perhaps most tragilcally in Russia, where they tried to go from tightly regulated Communism to a kind of laissez-faire Capitalism overnight and got horribly burned. And they ended up with the same kind of masters as before, because scum always rises to the top of the pond.
Re:WHAT! (Score:5, Insightful)
" I mean, just think of large corporations that avoid paying tax or buy cheaply from sweat-shops employing child-labour."
When all else is equal... /. about Chinese students taking exams instead of the ones who should. After many, many, MANY such stories over the years one can't help but develop a stereotype.
All companies have these kind of skeletons in the closet. Chinese companies simply seem to have some more on top of those which everyone else lovingly owns.
Lack of regulation might be a reason when seeing this internally (within the country) - and then again, when all else is equal... But here we're talking about international events, and that's where you see companies A, B, C playing by the rules and company D (Chinese, more often than not) trying to cheat its way in.
No more than a couple weeks ago there was a story here on
From fake $ITEM to cheating in competitions, China seems on top of the world as count of occurrences.
I googled "chinese cheating": got 22.6M results, top results are about exam cheating. :)
I also googled "americans cheating", got 14.8M results, top results are about marital cheating.
Of course, this might not mean much, but it's a start. Anyone wants to send a research grant my way?
Re: (Score:2)
I also googled "americans cheating", got 14.8M results, top results are about marital cheating. :)
Of course, this might not mean much, but it's a start. Anyone wants to send a research grant my way?
It's not even a start. In fact, using such incredibly naive and faulty methods brings into question the rest of your post and weakens your argument.
Re: (Score:2)
I googled "chinese cheating": got 22.6M results, top results are about exam cheating.
I also googled "americans cheating", got 14.8M results, top results are about marital cheating.
So, China, with 4.2 times the US population has 1.5 times more cheaters. I guess the irrefutable conclusion from your data is that Americans cheat 2.75 times more than the Chinese, right?
Re: (Score:2)
Also the amount of hits from US-based articles, being in English, should weight a lot less in comparison, and that's just one of the many factors to be considered.
It was a probably too inconspicuous joke about "lies, damn lies, and statistics". Next time I'll be more obvious about it, I promise.
Re: (Score:3, Insightful)
China is a developing nation, and they are still relatively new to the way companies play it in the West
Their GDP is twice as big as Japan's, the #3 economy in the world. At what point do they stop getting to play the "we're just a poor developing country, we can't be expected to follow the rules" card?
Re: (Score:1)
China is a developing nation, and they are still relatively new to the way companies play it in the West
Their GDP is twice as big as Japan's, the #3 economy in the world. At what point do they stop getting to play the "we're just a poor developing country, we can't be expected to follow the rules" card?
When their military is more powerful than that of the US?
Re: (Score:3)
he didn't say they were small or poor. Putting words into someone else's mouth and hitting them with it -- I think that is called a strawman.
developing nation
"A nation where the average income is much lower than in industrial nations..."
http://dictionary.reference.com/browse/developing+nation
Re: (Score:2)
Their GDP is twice as big as Japan's, the #3 economy in the world. At what point do they stop getting to play the "we're just a poor developing country, we can't be expected to follow the rules" card?
According to Wikipedia: A developing country "is a nation with an underdeveloped industrial base", whereas a developed country "has a highly developed economy and advanced technological infrastructure relative to other less industrialized nations". So, it is a matter of taste whether countries like India or China are still "developing", but the fact is that there is still a large proportion of the population that doesn't enjoy much in the form of mod coms.
I didn't claim that China is a poor developing natio
Re: (Score:2)
Absolutely agree. Many of the worlds problems (with both socialism and capitalism) come down to the fact that there are selfish dicks who ruin it for everyone.
Capitalism seems to do the best at harnessing these narcissistic dicks in a (sometimes) constructive way. Actually that is quite a remarkable achievement given that history is basically the story of how narcissist repeatedly oppress everyone else for their own benefit. Sadly it seems many people now believe it is the narcissist themselves, rather than
Re: (Score:1)
That's not unlike USA early in its industrial revolution. Remember The Jungle, by Upton Sinclair?
Chinese Hyper-competitiveness (Score:4, Interesting)
This reminds me of some Chinese ball screws (rotary to linear motion components) my company once ordered from Alibaba. The companies had pictures, drawings and put together quotes for parts but then delivered samples that were just totally totally useless. Some of these 'precision' parts looked like they had been made with a file. It just didn't make any sense that they would waste their and our time on such clearly incompetent products.
But when you go there you realise the problem. It is basically an economy in a state of hyper competition. There is so much competition that people will just try anything get ahead, completely oblivious to the wider problem or goal they are trying to solve. You can see that in how the government had to rationalise the solar industry because nobody could make any money. They are just really really crazy competitive.
The trouble though is that there are now many good Chinese engineers who know what they are doing but are still hyper competitive. I really don't know how us westerners with our 40hr work weeks, healthcare and pensions are going to eventually compete with that until we too are faced with the desperation of trying to escape from abject poverty along with 1 billion other people.
Re: (Score:3, Insightful)
people will just try anything get ahead, completely oblivious to the wider problem or goal they are trying to solve
Their goal is to make money, even if it means screwing the customer. Their culture is such that if you let them screw you it's your own fault and you should be mad at yourself, not the person who cheated you.
I really don't know how us westerners with our 40hr work weeks, healthcare and pensions are going to eventually compete with that until we too are faced with the desperation of trying to escape from abject poverty along with 1 billion other people.
The key word there is "eventually". Long term "us Westerners" are leading and will continue to work together to make real progress instead of pretending to be ahead by faking results and cheating on contests and other things like SAT tests.
Re: (Score:2, Insightful)
"China has no such internal conflict."
You must be kidding, or not know anything about China.
Re: (Score:1)
Their culture is such that if you let them screw you it's your own fault and you should be mad at yourself, not the person who cheated you.
So their culture lauds psychopathic and extremely selfish behavior?
I guess complaining about them being psychopathic selfish assholes won't fix the problem will it?
But I bet they'd get the message if we dropped a couple nukes on them, or blockaded all trade in and out for them, or sabotaged their computer systems in their entire country, or maybe if we started assassinating their top leaders, or destabilizing their country... Whatever it takes for us to get ahead, y'know? It would be their fault for pis
Re:Chinese Hyper-competitiveness (Score:5, Interesting)
You need the buyers to stand up to that kind of crap. These guys screwed (sorry) you once, black list them. I had a frustrated guy in procurement say "You could make tons of money in x business by just doing what you say you are going to do." When it came down to it, he was not always allowed to make the smart decision and go with a reliable company's bid if it wasn't close to the bottom.
Part of the problem we see in China is that they spawn new companies like rabbits. One will eventually go under doing shady stuff only for the same people to come back, selling the exact same crap under a different name. We had this happen and you could look at the product and see that they were using the exact same molds, same defects.
Re: (Score:3)
Except they tend not to be major companies with government backing.
Let's bail out the banks... then bail out the banks again!
Let's bail out the automakers... then bail out the automakers again!
Get real.
Re: (Score:1)
Something that makes it more interesting is infrastructure sharing. In China, you'll often get a plant that was set up to produce reliable parts for some global corp where someone who works at the factory sets up an independent company that comes in at night with lower-grade source materials and uses the exact same facilities with a different staff. Kickbacks then go to the management of the original company.
As a result, you can get precision-crafted products that look just like top-of-the-line materials,
Chinese cheat (Score:5, Interesting)
People growing up under oppressive governments have much fewer problems with cheating — because cheating government is a fair game. It rubs off — and the attitude is quickly extended to non-governmental institutions large and even smaller ones.
This is not "racism" — ex-Soviets like myself often have the same problem... A cheating Western student fears (or used to fear) the shame of being exposed. A Chinese — or a Soviet — fears merely getting caught. Like a speeding ticket — there is no shame in driving fast, only in being stopped by "the bear".
China today uses drones to catch cheaters [engadget.com] — America had not felt the need for such measures. Perhaps, it was a foolish attitude, because we the immigrants bring all our traits [slashdot.org] to the "wonderful tapestry of diversity", not just the good ones...
Anybody dealing with Chinese companies (or Russian ones, if you can find any), ought to be careful and not depend merely on trust.
Re: (Score:3)
Yeah... because two wrongs make a right.
The rules of life are not "whoever dies with the most toys wins", but it seems to me that people who think cheating is a good idea often seem to live their lives as though that were the case.
Striving to live one's life with integrity may not be easy, but then nothing that is really worth doing ever is.
Re:Chinese cheat (Score:5, Insightful)
You know, when your parent society doesn't value honesty, and everybody around you is cheating ... you're a fool to think there's any value in being that one guy who says "gee, I should be honest here".
In situations in which it's a liability to be honest, only suckers are honest.
And in governments who have spend decades saying "there is no higher power than the state", if the state is rampantly corrupt, "integrity" is a relative term.
Give it a few more years, and you'll discover that integrity in America is a much more malleable concept than you realize -- in fact, it's probably already there.
The mentality of "it's OK as long as I don't get caught" isn't a new thing to humanity.
Re: (Score:3)
Yes, well.. Truth be told, cheating and being dishonest are usually not long term strategies for success under any system. You cheat in school? You will pay later when the next course comes along that uses the material you don't know. Cheat again and you just increase your chances of getting caught. One cheat leads to another, then more, usually bigger and bigger ones until you get caught and pay the piper for cheating.
In my world view, the reward in doing the right thing over the long haul is in knowing
Re:Chinese cheat (Score:4, Insightful)
Re: (Score:1)
Meh, whatever ... go cram your high horse up your ass.
I'm not saying everybody should do it, or that I think it's right.
I am saying human nature is generally such that if everybody else is doing it, and you might not fare as well by not doing it, you'll probably find yourself doing it or suffering for not having done so.
The world is a cruel place, and doesn't always reward good behavior. When you see bad behavior being rewarded around you , you say fuck it and do the same.
But, by all means, continue to bel
Re: (Score:2)
Using the Ethics Scoreboard [ethicsscoreboard.com] as a reference point, I find that the only real difference between the Futility Illusion (point #10) and this argument is that where the former typically tends to imply that it is all but inevitable that someone else will behave badly even if one tries to behave ethically, this argument actually fallaciously attempts to strengt
Re: (Score:2)
As for suggesting that one is somehow foolish to try and live with integrity when others are cheating.... Well thats just a variant on an ad hominem, called Poisoning the Well.
You're foolish to let people who are cheating get ahead of you because you're unwilling to cheat. That's not getting you anywhere, and it's just rewarding them and punishing you.
If you want to be foolish, because there are things more important to you than making shrewd decisions, so be it. I agree. But I'm not hungry, in fact I'm eating lunch right now.
Re: (Score:2)
Re: (Score:1)
Whatever benefits me is what I will do and screw everyone else.
isn't much different from
Whatever benefits me is what I will do and if others get screwed then that's a shame.
which isn't much different from
Whatever benefits me is what I will do, but I'll put the "screw them" option in last... but it's still an option.
How about just do the right thing?
Re: (Score:2)
I was pointing out what I find to be a considerable benefit to living with integrity as not necessarily an incentive for anyone to do so, rather I was suggesting that that the notion that there is an alleged lack thereof as an disincentive to live honestly was mistaken.
And "screwing them" isn't even really an option in how I approach life, last or otherwise, except to the extent that those whose paths cross mine and who might live more dishonestly may possibly find themselves getting judged as less trust
Re: (Score:1)
Yes it happens everywhere, but some places are more susceptible to obvious graft and cheating than others. In the USA, for example, it would be hard to argue that there isn't a certain culture of graft in government... but it's mostly a legally acceptable variety called "campaign contributions." Unfortunately the people making the rules have made a system where they're exempt many of them.
The thing is, what might not be unexpected in certain levels in North American society permeates more levels in some oth
Re: (Score:1)
Some times they certainly do. Consider — killing a man is wrong, right? But adding the second wrong of killing an attacker trying to kill you makes a right...
Likewise, cheating an entity, that cheats you may be perfectly ethical — when you can not, as is the case with governments, to simply stop dealing with the cheater.
Once that attitude is developed, it becomes hard to shed and is often applied even when unjustified and, indeed, quite unethical.
What I
Re: (Score:3, Interesting)
The rules of life are not "whoever dies with the most toys wins"
But the rules are "whoever doesn't eat loses". Historically, this was not about accumulating toys, but basic survival. During the Great Leap Forward, 30 million Chinese starved to death. There were another few million deaths from starvation and violence during the Cultural Revolution. Both of those were caused by government abuse and incompetence. Even today, China has a hereditary class system [wikipedia.org], that denies hundreds of millions of people access to housing, healthcare, and even the right to send their c
Re: (Score:2)
Re: (Score:2)
Well, my point was to address the comment that the idea that integrity should be valued is allegedly somehow "western" notion.
I didn't mention Christianity because there may be some disagreement about it being "western", arguably having evolved into the state it is recognized today mostly in Western Europe, although certainly its foundations are at least as old as Islam.
Re: (Score:2)
I didn't mention Christianity because there may be some disagreement about it being "western"
Then you shouldn't have included Islam either. Senegal is Islamic, is further west than any part of Europe, and is completely in the western hemisphere.
Re: (Score:2)
Re: (Score:1)
Yes, but neither of those examples are Eastern religions.
Or easier still (Score:5, Insightful)
They'll just go in and steal the research from another competitor and call it their own. Cheating and espionage are familiar bedfellows.
Nothing new (Score:3)
Such cheat have been used for years in the field. Before ImageNet, there was the Pascal VOC challenge with about the same rules, and I'm pretty sure all winners were optimizing the hyperparameters their submission on the test dataset.
Seriously, as long as computer vision benchmark are based on a single train/test split, there will be such abuses. If there were several splits with meaningful statistics computed on it, I would be less worried by the overfitting you get by optimizing the hyperparameters.
But hey, you're never gonna make it to CVPR without tunning your method so as to fool reviewers that it performs much better than the state of the art. 0.1% for a good idea, 99.9% for engineering tricks.
Banned for 1 year? (Score:2, Insightful)
Maybe that's appropriate punishment for children, but these are professional scientists. The only reason nobody has the brass to ban them for life is because their country owns us.
Re: (Score:2)
The only reason nobody has the brass to ban them for life is because their country owns us.
Nobody wants to ban them for life because even if they learn their lesson, we won't benefit from their competition in the contest. Don't throw out the baby.
We know what Baidu is, thankyouverymuch. (Score:5, Informative)
Re: (Score:2)
The headline is either misleading or completely clueless.
Almost as clueless as thinking you won't get caught cheating on a computerized test that logs everything... yet, not quite.
Re: (Score:2)
China cheating?? (Score:3)
Surely you jest!
Priorities (Score:1)
China is a fast-moving, up-and-comer nation in the modern sense, but they lead the way in air pollution. Some mock our nation's EPA, but you can thank them, and other local entities like them, who put air and water quality above corporate profits, despite the many complaints from the largest abusers and other overly friendly corporate shills and lackeys. Allowing businesses to run amok in the name of a few low-paying jobs and letting them skip out on paying a fair share in taxes is how these things happen.
Baidu Team's Apology Appended to Official Notice (Score:4, Insightful)
Message from the team in question:
Dear ILSVRC community,
Recently the ILSVRC organizers contacted the Heterogeneous Computing team to inform us that we exceeded the allowable number of weekly submissions to the ImageNet servers (~ 200 submissions during the lifespan of our project).
We apologize for this mistake and are continuing to review the results. We have added a note to our research paper, Deep Image: Scaling up Image Recognition, and will continue to provide relevant updates as we learn more.
We are staunch supporters of fairness and transparency in the ImageNet Challenge and are committed to the integrity of the scientific process.
Ren Wu – Baidu Heterogeneous Computing Team
So, while they deserve the year ban, the apology is nice. It's a shame we can never know what results a fair competition could have yielded ... and an even bigger shame that the media misreported Baidu as overpowering Google. I suppose the damage is done and the ILSVRC has made the right choice.
...
Perhaps I'm misunderstanding the classification problem but why isn't this run like most other classification problems (like Netflix and many other data challenges) where you get ~80% for training and the remaining 20% are held back for the final testing and scoring? Is the tagged data set too small to do this? Seems like wikimedia would contain a wealth of ripe public domain images for this purpose
Re: (Score:2)
Can we teach machines to cheat? (Score:1)
LOL. Anybody surprised? (Score:2)