AtlantaStartupWeekend 

Blog Posts

blog posts

Bookmarks:

The Morning Pre-Run Rundown

As I stall before I head out the door for an hour run in the cold, dark mountains outside my house, I thought I'd share some of the interesting stuff I read this morning with you.

Forrester Chief on What Not to Cut in a Downturn: I mostly include this since it's so entertainingly short and substance-free.  It links to a post of George Colony's blog titled Why this tech recession will be different which is still short, has a little more substance, but seems painfully obvious.  Maybe people will cut their Forrester spending in the downtown (oh, cynical me.)  I hope no one at Forrester is using social media or any keyword alerts that catch this blog or else I'll be in trouble with my friends at Forrester.

Where is Dubai? Jeff Jarvis has an awesome essay on his trip to Dubai.  He's also got some pretty pictures in the post.

Five Minutes with Angel Investor Dave McClure: I've done a few investments with Dave and think he's dynamite.  I went to Startup2Startup a few months ago when I was in the bay area and had a blast.  Good stuff.

Harvard: 'Nothing Is Fucked, Dude': Drew Faust, Harvard's president, writes a long email on the state of Harvard given the economic downturn titled "Harvard and the economy".  In it she acknowledges the forecast that many college endowments will be down as much as 30% this year due to investment losses.  Eek.  Now's a good time for my MIT friends to do some extra hacks on Harvard's campus since there will be fewer security guards running around.

And now for some news from the world of companies that I've invested in.

How and Why We Made Glue: Alex Iskold has a long post up explaining the design choices behind the recent release of Adaptive Blue's product.  The product - Glue - is dynamite and the explanation of the design choices is really interesting.

Solution Spotlight: Soup.io is now using Gnip: Adoption of Gnip marches on day by day.  Subscribe to the Gnip blog to see how more companies are using it.

Look Out Google Site Search, Lijit Says It's Right On Your Heels: ReadWriteWeb analyzes Lijit's Widget Statistics Revivial 2.0 and comes up with some interesting thoughts.  Even more interesting (at least to me) are the kind and generous words they have about Lijit's service.  If you have a blog but don't yet use Lijit for your blog search engine, you are missing out.

Ok - enough stalling.  Time to get dressed and go running.

The Morning Pre-Run Rundown

As I stall before I head out the door for an hour run in the cold, dark mountains outside my house, I thought I'd share some of the interesting stuff I read this morning with you.

Forrester Chief on What Not to Cut in a Downturn: I mostly include this since it's so entertainingly short and substance-free.  It links to a post of George Colony's blog titled Why this tech recession will be different which is still short, has a little more substance, but seems painfully obvious.  Maybe people will cut their Forrester spending in the downtown (oh, cynical me.)  I hope no one at Forrester is using social media or any keyword alerts that catch this blog or else I'll be in trouble with my friends at Forrester.

Where is Dubai? Jeff Jarvis has an awesome essay on his trip to Dubai.  He's also got some pretty pictures in the post.

Five Minutes with Angel Investor Dave McClure: I've done a few investments with Dave and think he's dynamite.  I went to Startup2Startup a few months ago when I was in the bay area and had a blast.  Good stuff.

Harvard: 'Nothing Is Fucked, Dude': Drew Faust, Harvard's president, writes a long email on the state of Harvard given the economic downturn titled "Harvard and the economy".  In it she acknowledges the forecast that many college endowments will be down as much as 30% this year due to investment losses.  Eek.  Now's a good time for my MIT friends to do some extra hacks on Harvard's campus since there will be fewer security guards running around.

And now for some news from the world of companies that I've invested in.

How and Why We Made Glue: Alex Iskold has a long post up explaining the design choices behind the recent release of Adaptive Blue's product.  The product - Glue - is dynamite and the explanation of the design choices is really interesting.

Solution Spotlight: Soup.io is now using Gnip: Adoption of Gnip marches on day by day.  Subscribe to the Gnip blog to see how more companies are using it.

Look Out Google Site Search, Lijit Says It's Right On Your Heels: ReadWriteWeb analyzes Lijit's Widget Statistics Revivial 2.0 and comes up with some interesting thoughts.  Even more interesting (at least to me) are the kind and generous words they have about Lijit's service.  If you have a blog but don't yet use Lijit for your blog search engine, you are missing out.

Ok - enough stalling.  Time to get dressed and go running.

Data Debasement

Last week I was in Boston to moderate a panel at the MIT Technology Review's Emerging Technologies Conference -- one of those tech shindigs so expensive I can only attend as hired help. My panel was on parallel computing and it produced this column and another I'll file early next week. This week is about databases and next week is about threads. Isn't this a grand time to be a nerd?

Thanks in part to Larry Ellison's hard work and rapacious libido, databases are to be found everywhere. They lie at the bottom of most web applications and in nearly every bit of business software. If your web site uses dynamic content, you need a database. If you run SAP or any ERP or CRM application, you need a database. We're all using databases all the time, whether we actually have one installed on our personal computers or not.

But that's about to change.

We're entering the age of cloud computing, remember? And clouds, it turns out, don't like databases, at least not as they have traditionally been used.

This fact came out in my EmTech panel and all the experts onstage with me nodded sagely as my mind reeled. No database?

No database.

Parallel computing used to mean scientific computing, where hundreds or thousands of processors were thrown at technical problems in order to solve them faster than Moore's Law might otherwise have allowed. The rest of us were relying on rising clock rates for our performance fix, but scientists -- scientists with money -- couldn't wait so they came up with the idea of using multiple CPUs to solve problems that were divided into tasks done in parallel then glued back together into a final result. Parallel computing wasn't easy, but sometimes that was the whole point -- to do it simply because it was so difficult. Which is probably why parallel computing remained a small industry until quite recently.

What changed was Moore's Law put an end to the clock rate war because chips were simply getting too hot. While faster and faster chips had for the most part linear performance increases along with cost and power consumption decreases, the core temperature inside each microprocessor chip was going up at a cubic rate. Back in 2004 Intel released a chart showing that any clock speed over 5 GHz was likely to melt silicon and Moore's Law would, by 2010, make internal processor temperatures similar to those on the surface of the Sun!

For those, including me, who think that's pretty darned hot I'll point out one of my astronomer readers immediately had to mention that the Sun's chromosphere is actually much hotter than the surface. Forgive him, he means well.

Faced with this absolute thermal performance barrier, Intel and AMD and all the other processor companies had to give up incessant clock speed increases and get us to buy new stuff by putting more than one CPU core in each processor chip can. Now chips with two and four processor cores are common and Intel hints darkly that we'll eventually see hundreds of cores per chip, which brings us right back into the 1970s and '80s and the world of parallel computing, where all those principles that seemed to have no real application are becoming very applicable, indeed.

And that's exactly where databases start to screw up.

Bob Lozano, chief visionary, evangelist, father-of-eight (same woman) at Appistry came up with the first database example I'd heard and it was eye opening. Appistry (I've written about them before -- it's in the links) specializes in distributing what would normally be mainframe applications across tens, hundreds, or even thousands of commodity computers that act as one. If a government agency wanted to do meter-scale analysis of very high-resolution images from spy satellites, it might use Appistry, I'd imagine (just a guess, of course).

The database example Bob used at MIT was of an unnamed customer that is a credit card transaction processor. Their core application -- the processing of credit card transactions -- was happening on IBM Z-series mainframes (the biggest of big iron) and the client wanted to port the whole mess to commodity PCs.

Google did it, why not them?

But the first time they tried it at Appistry, it didn't work.

"We hired a technical team that had done similar applications before so they started with replicating the mainframe architecture on the commodity computers, database and all," Lozano recalled. "But when they finished it wasn't appreciably faster than the mainframe it replaced. Even worse, it wouldn't scale. So we fired the team and started over."

The problem, it turned out, was in the database.

The way the original mainframe application functioned was by first receiving transactions, writing them to the database, reading them from the database, then doing the actual processing before writing them again back to the database. That's read-write-read-process-write.

The second time through the Appistry team tossed the database, at least for its duties as a processing platform, instead keeping the transaction -- in fact ALL transactions -- in memory at the same time. This made the work flow into read-process-write (eventually). The database became more of an archive and suddenly a dozen commodity PCs could do the work of one Z-Series mainframe, saving a lot of power and money along the way.

If this sounds like a risky way to do business -- not writing the data to disk until things slow up enough -- remember that's the way Google runs its search engine and why it is so darned fast. Google has THE ENTIRE INTERNET IN MEMORY AT ONCE. If the application slows down they just add more hardware.

This is good news for cloud computing and bad news for mainframes, because systems like Appistry and its competitors (there are several) are going to eventually bury the mainframe by putting the "cloud" into cloud computing. Suddenly a Storage Area Network with a relatively weak database controller is good enough for archiving while the parallel or even massively parallel cloud does the real work.

Later this month Microsoft will announce a cloud version of Windows Server and it will be very interesting to see how it handles database integration and dis-integration.

The database problem is much more than just slow reads and writes. Relational databases also create false dependencies between pieces of data. Dependencies of any kind break parallelism, and therefore make an application hostile to commodity platforms. That is, if one chunk of data (A) is dependent on another chunk of data (B), then no work can be done on A until all work on B is complete. If the dependency is real, like when A and B are both withdrawals from the same bank account, then there are hacks we can try like one I will describe in my next column, but most programmers just choose to have a cup of coffee and wait for B to finish.

But if these are withdrawals from different bank accounts, or maybe even different banks, then no true dependencies exist. Unfortunately, if all of this has been stored in a single relational database then we unintentionally create a false dependency, since that database can only handle a fairly limited amount of items concurrently -- we've created a bottleneck that will choke the application.

While the database guys are busy figuring out how to add more and more concurrency internally, in reality when you take a few steps back and think of a large set of commodity boxes all executing a single data munching app, then no matter how sophisticated we get, the relational database will still effectively be a single thread to that app.

A traditional response is to pour dollars into the data tier, buy faster, more concurrent SANs, better interconnects, and bigger database servers. That works up to a point and pleases Sun and IBM no end.

Somewhat more helpful though are data grid products like Coherence or eXtreme Scale (IBM), or Appistry's Fabric Accessible Memory (FAM). For many applications those can take more of the lifetime of each chunk of data and keep it in memory in the middle tier -- hopefully on the same boxes where this large data-munching app resides. But this still doesn't completely solve the problem because there remain limits on how far the relational database can scale.

Here's how Google attacks this problem, which goes beyond simply keeping the entire Internet in memory. The problem, of course, is that you can't keep the entire Internet in memory in every server because then you'd need more memory chips than even Google can afford to buy.

To scale the Google search service, then, they figured that many large problems did not intrinsically require doing actions one at a time. But Google first had to free itself of the false dependencies. So they coined the term MapReduce and created both a set of operations and a way to store the data for those operations natively, all while preserving the natural independence that is inherent in each problem, building the whole mess atop the remarkable Google File System, which I'll cover some other day.

Google led the way but many other companies have followed suit, opening doors to a wide range of new ways of thinking about large-scale data manipulation. Suddenly there are different ways to store the data, new ways to write applications, and new places (thousands of cheap boxes) to run such applications.

What this does for Larry Ellison and his libido is a great question, because it looks like he's bought up most of the traditional database-centric software industry just in time for it to be declared obsolete.

Sorry Larry.

Gnip 2.0 Launches, With A Business Model

Gnip, the guys that are helping move data around from one social network to the next, launched v 2.0 of the service tonight.

The new version of the service allows data consumers (services like Plaxo that take data from other services, like Twitter, Friendfeed, Digg, Delicious, etc.) to have data from requested users pushed to them. It’s no longer “Hey, TechCrunch just tweeted. Go query the API to get the data.” Now it’s “TechCrunch just tweeted - here’s the data.” Data consumers are no longer required to build pollers for any of the publishers pushing data into Gnip, they just give Gnip an endpoint and they push the data to them in real time.

Data consumers can get complete public data streams for Twitter, Digg, Delicious, Six Apart and others without ever visiting those sites or accessing their individual APIs, subject only to the terms of service of those services. And this data can be gathered via a REST-based API or the newly launched XMPP support.

Gnip also added a number of filter options to allow data consumers the ability to create rules based queries based on tags, keywords, etc.

Gnip’s business model is freemium - lots of data for free and commercial data consumers pay when they go over certain thresholds (non commercial use is free). The model is based on the number of users and the number of filters tracked. Basically, any time a service is tracking more than 10,000 people and/or rules for a certain data provider, they’ll start paying at a rate of $0.01 per user or rule per month, with a maximum payment of $1,000 per month for each data provider tracked. For now billing is turned off and the service remains completely free. Thirty to sixty days from now, people will start to pay.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

Charlie Rose on the Financial Crisis

Charlie Rose continues to do, far and away, the most thoughtful and lucid work on the ongoing financial crisis. The following conversation from last night's show is another good example. It includes my friend Nouriel Roubini, who came straight over from Charlie's show to a Money:Tech dinner I was hosting in New York.

Impossible

Right now, there's a CEO standing in front of his 85-person start-up at an all-hands meeting and he's saying, "In the next 90 days, we need to do the impossible".

The particular version of impossible doesn't matter. What matters is that everyone in the room is shocked when he says it. You can tell by the intensity of the silence.

"We're going to what in the what?"

What gives this guy the right to ask the impossible? Sure, he’s the CEO, but does that mean he gets to stand in front of the room and ask the team to build a levitation machine?

Yeah, it does.

However, this does not mean the CEO isn’t screwing up.

Asking for the impossible is an advanced management technique and it’s one that is particularly abhorrent to engineers. They are very clear on what is and isn't possible because they're responsible for building and measuring all the possible. When you ask an engineer to do the impossible, they often laugh in your face not only because they think it's an absurd, irrational request, they also have the data to prove it.

Yet, given this irrefutable data, I still want you to consider this request. There is an upside to pulling off the impossible. Not only is it a great morale booster, it can also be incredibly profitable, because all your competition thinks the impossible is, well, impossible. Better yet, WHO DOESN’T WANT A FLUX CAPACITOR?

There are three measurements to take with regard to your CEO and his request when the team has been asked to do the impossible. These measurements aren’t going to help you pull off the miracle, but they will help you size the impossibleness.

A Hint of an Insane Plan

First, let’s figure out if your CEO is insane. Listen carefully to the actual request. If your CEO is standing in front of the engineering team asking you to transform lead into gold, you should grin, nod, and start mentally editing your resume, but don’t bolt from the room just yet. Now, if he's asking you to reduce your release cycle from 90 days to 10, you can let yourself be shocked, but be relieved by the fact that you're not being asked to perform matter transmutation.

There’s a subtle difference between insane and impossible.

You should respect your gut when that internal “he’s insane” flag starts waving, but that doesn’t mean you should stop listening. There’s more data to gather and there are times where an insane approach might be the right thing.

Our next assessment has to do with legwork. Has your CEO done any preliminary work to actually figure out whether the impossible idea is achievable? What is his strategic intuition about this crazy idea? Is he able to articulate, however vaguely, why this idea is a good idea for the company, and how you might pull this off? You're not looking for a definite plan, more the strategic broad strokes, a point from which the managers can begin sketching in the details.

A word of warning: there are managers and executives out there who can pitch the impossible on confidence alone. They need no intuition or evidence regarding feasibility to get their teams’ buy-in, and while these chutzpah-laden individuals sure are inspiring, you should trust that nagging feeling that shows up later when you're driving home, the high fades, and you're left with a strategic emptiness. That emptiness is the practical result of the CEO’s request lacking everything but confidence. The absence of some thread of an idea about how you're going to do the impossible, and you might be screwed.

The lack of a glimpse of a plan beyond the charisma translates to a lack of hope.

Skin in the Game

Next, you want to figure out how much skin your CEO has in the game. How much of the company is he betting on this request? If this is a bet the company decision, I’m comforted by the fact that he’s backing this impossible request up with his job. He knows that failure means everyone is looking for a new gig. That’s motivation.

If the request is smaller, if this is a bet the department request, well, the risk is more localized. The cost of failure will likely be born by the senior guys and gals running the show. I’m not suggesting the CEO thinks any less of the importance of this impossible request, but, trust me, he knows that it’s not necessarily his job on the line if the team blows it.

What you’re assessing here are two things: size of the request and level of executive commitment. Having a gut feel for these two things is often a moot point. Depending on your seat on the org chart, you might not even have a chance to choose whether you’re saddled with the impossible. However, developing this swag out of the gate means when the impossible hits the fan you can be one of the first to act.

The Importance of Respect

The glimpse of a plan and confidence. These two fuzzy mental assessments are in play when deciding to ask the impossible, but there is one more that needs to be considered.

Remember, this is an impossible request. This isn't, "Hey, can you fix these 10 bugs by Friday?” It's "Hey, can you rewrite this major component in half the time it took you to write it the first time?" Forget whether it's remotely feasible. Forget whether the confidence is oozing out of every pore of your CEO. You’re not going to be convinced, and more importantly, you're not going to engage if you don't respect the person who is asking you to do something

Financial rewards, promotions, IPOs, promises of future interesting projects. All of these incentives matter and can be used to light a fire under a team, but an individual's decision to engage in the impossible starts with the question, "Do I respect this person enough to tackle the impossible?"

There's a book to be written about how to build respect in an organization. My brief advice is, when you are asked the impossible, carefully consider every hard request already made of you. Does he ask the impossible every month? Every Monday? Does he follow up on his impossible requests or does he expect you to run with them? Have we ever successfully completed an impossible request? Is he there at 3am on Sunday morning with everyone else, looking like he hasn't shaved in a week?

I don't know how many impossible requests you get, but I do know that frequent impossible requests result in an erosion of respect and a decaying of credibility. And that means when the CEO is standing up in front of the troops asking them to perform magic all they're thinking is, "This crap again?"

What He Really Wants

Nothing I've described is concrete. Nothing I’ve described is going to placate your initial intense, negative engineer reaction when your CEO asks you to do something utterly absurd and irrational.

It gets worse… I mean better.

There are times when your leadership should be unencumbered by your version of reality. There are times when it's important that your CEO isn't intimately familiar with a product space or lifecycle. Day to day, doing business requires reasonable expectations and an adherence to plans, but those things actually prevent the extraordinary from occurring. The extraordinary requires a catalyst like an impossible request.

What's important when the CEO asks for the impossible is that he’s pushing the definition of possibility for what the team can accomplish. Maybe your CEO only has an idea, and can only feel the possibility in what he's asking, but it’s not his job to make it all happen. That’s where you come in. You're the person responsible for transforming the feel, the intuition, the glimpse of a plan, and the confidence into knowing and doing.

You’re the one who is actually responsible for delivering the impossible, and all I’m asking is that you consider the request, because agreeing to engage in the impossible shatters normality and ignores fears and I love that.