Harvard/MIT Student Creates GPU Database, Hacker-Style 135
First time accepted submitter IamIanB writes "Harvard Middle Eastern Studies student Todd Mostak's first tangle with big data didn't go well; trying to process and map 40 million geolocated tweets from the Arab Spring uprising took days. So while taking a database course across town at MIT, he developed a massively parallel database that uses GeForce Titan GPUs to do the data processing. The system sees 70x performance increases over CPU-based systems, and can out crunch a 1000 node MapReduce cluster, in some cases. All for around $5,000 worth of hardware. Mostak plans to release the system under an open source license; you can play with a data set of 125 million tweets hosted at Harvard's WorldMap and see the millisecond response time."
I seem to recall a dedicated database query processor that worked by having a few hundred really small processors that was integrated with INGRES in the '80s.
Re:I'm not a computer scientist, and... (Score:4, Insightful)
Sprinters can run really fast. So, if speed is important in other sports, why aren't the other sports full of sprinters? Because being good at one thing doesn't mean you're well-suited to do everything. A sprinter who can't throw a ball is going to be terrible at a lot of sports.
Re:I'm not a computer scientist, and... (Score:2, Insightful)
Yes, it is that bad. Not only is it extremely platform-specific, the toolchains are crap. We're just now transitioning from "impossible to debug" to "difficult to debug".
Re:I'm not a computer scientist, and... (Score:3, Insightful)
Parallelization is not why GPUs are fast, its a side effect of rendering pixels, nothing more.
GPUs are fast because they do an extremely limited number of things REALLY REALLY fast, and when you're doing graphics ... well guess what, its all pretty much doing those few things the GPU does well over and over again, per pixel (or vertex). They are parallelized because those simply, super fast processors are also small from a chip perspective, so stuffing a ton of them on the chip so it can do many pixels in parallel works, again, because all those pixels get treated the same way with a very limited number of well known operations performed on them.
They are not replacing CPUs because something like a simple if statement doesn't pause one processor, it pauses them ALL, and then top it off with the GPU being absolutely horrible (speed wise) at dealing with an IF statement. In shaders, you can get by with an if on a uniform because it only has to be calculated once and a decent driver can optimize the if away early on be for sending it to the cell processors on the CPU. Do IFs on an attribute (say a vertex or texture coord) and watch your GPU crawl like a snail.
Parallelization in GPUs is a direct result of the fact that they perform the same task on massive arrays of data. Since the code works on individual cells in the array individually, there is no 'race' condition possibility in the code, so its ready to run concurrently. Adding a new shader cell effectively gives you more speed without any sort of programmer effort what so ever.
The reason these parallel cells can work together so fast is also because the silicon works in lock step. (thats why IFs or attributes kill performance). Basically each line of the shader program executes side by side on all the shader cells at once. This makes all sorts of neat silicon based performance tricks possible.
Where you get screwed however is those IFs (All branching instructions really) because if any one shader cell has to run a branch of code, they ALL run the code, and then just discard the results. So when you write branching code in a shader, you are almost certainly going to run every code path provided if you use the wrong data for your branch.
Re:I'm not a computer scientist, and... (Score:4, Insightful)
I think you totally missed his point -- tin whiskers on your circuit board? Blown caps?
The fact that 9 women can have 9 babies in 9 months for an average rate of 1/mo, does not disprove the assertion 9 women cannot have __a__ (i.e. a single) baby in one month. You're talking about something totally different and being awfully smug about it to boot.
Re:Q: Whats better than a GPU database? (Score:3, Insightful)