Heartbleed (OpenSSL vulnerability)

Kind of surprised someone hasn't already made a topic about this one. I know a lot of folks here run Linux at home and/or at work.

Anyways, if you're running OpenSSL anywhere, you should definitely check to see if you're running a version with the bug, and update if so. It's a nasty one.

Most of the major Linux distros came out with updates yesterday or this morning. If your router is running some sort of embedded linux, you may want to keep your eye open for a firmware update, especially if you allow remote management.

http://heartbleed.com/

Test a server for the issue:
http://filippo.io/Heartbleed/

AWS's ELB is affected:
https://forums.aws.amazon.com/thread.jspa?messageID=535168

Edited for more info and links.

Had to fix a few sites today - fun stuff.

Between this and our platform going down today, I don't know if my Ops guy is going to live through the night.

We've lucked out a bit. We primarily use FreeBSD for own infrastructure as well as for our managed service clients. We've haven't had the time to really start making a move to FreeBSD 10, so are mostly on 9.2 with a bit of 8.4 here and there still. The base openssl on those versions isn't subject to this vulnerability, while it is in 10. We also have a lot of managed PFSense boxes ... where again we're a bit behind where we intended to be upgrade wise, and are still mostly on 2.0 ... 2.0 is not vulnerable while the latest versions are.

There was a posting somewhere of the 'top 1000 sites' and their vulnerability status (I didn't see what criteria were used to make them the top 1000). A brief glance through that list showed one current client who was not vulnerable, and another who had recently left us in favor of Amazon who is. That was gratifying.

absurddoctor wrote:

There was a posting somewhere of the 'top 1000 sites' and their vulnerability status (I didn't see what criteria were used to make them the top 1000). A brief glance through that list showed one current client who was not vulnerable, and another who had recently left us in favor of Amazon who is. That was gratifying.

There's this one that claims to be based on the top 10,000 Alexa ranked sites.

I find that this is the rare technical issue where I'm not sure what to do on the client side. Anything? I've seen it stated that most browsers use a different SSL library, so the client-side exposure may be less? I should probably check on my (small-time) web hosting too, I assume.

Remember that you should revoke and reissue certificates that were used on any OpenSSL-enabled site. Otherwise, bad guys (including the government) may be able to impersonate your site, because they could have invisibly grabbed the server's private key without any visible log or evidence of intrusion. And they will definitely be able to surveil all traffic on your site until you generate a new key.

This is a BIG deal. I'd be very interested to know who, exactly, introduced this bug, because it's a world-class security disaster. In our brave new world where the US spy agencies have declared everyone, including US citizens, to be enemies, this is a tremendously convenient error.

Malor wrote:

This is a BIG deal. I'd be very interested to know who, exactly, introduced this bug, because it's a world-class security disaster. In our brave new world where the US spy agencies have declared everyone, including US citizens, to be enemies, this is a tremendously convenient error.

It's in the OpenSSL git repo. It was written by one of the authors of the TLS heartbeat RFC, and merged by one of the core devs.

http://git.openssl.org/gitweb/?p=openssl.git;a=commitdiff;h=bd6941cfaa31ee8a3f8661cb98227a5cbcc0f9f3

A friend of mine pointed out the time:
Date: Sat Dec 31 23:00:36 2011 +0000

Don't party and merge code at the same time!

Gremlin wrote:
absurddoctor wrote:

There was a posting somewhere of the 'top 1000 sites' and their vulnerability status (I didn't see what criteria were used to make them the top 1000). A brief glance through that list showed one current client who was not vulnerable, and another who had recently left us in favor of Amazon who is. That was gratifying.

There's this one that claims to be based on the top 10,000 Alexa ranked sites.

I find that this is the rare technical issue where I'm not sure what to do on the client side. Anything? I've seen it stated that most browsers use a different SSL library, so the client-side exposure may be less? I should probably check on my (small-time) web hosting too, I assume.

There isn't a lot you can do at this point. If you are going to visit a site using an https connection, you can use the test link above first to see if they are still vulnerable. If not, it doesn't meant they were not at some point ... but also no one knows if this was ever actually used by any bad guys before now. If the site is vulnerable, don't send any login or other information that you might care to it.

Malor wrote:

Remember that you should revoke and reissue certificates that were used on any OpenSSL-enabled site.

Seems worth mentioning that caveat that not all versions of openssl were vulnerable, and there is still a lot of 0.98 out there in the wild that will not need to worry about revoking certificates. For everyone else, this will probably be the most annoying and time-consuming aspect (minus actually being compromised).

Total layman here, I'm going to look over the the links (I've just read the thread, this is the first I'm hearing of this thing.) So… Does this mean that if I use an https sites (such as gmail) that I should change my password, just to be sure? At first blush this looks pretty freaking intense.

Well, other than the PITA factor, it's always a good idea to change passwords.

edit: Ars Technica is seeing some examples of stolen passwords. It's definitely been exploited. Full password changes everywhere would probably be an excellent idea.

It turns out that there were two openssl sa's, the second of which does effect older versions as well. Fortunately, while it will require some updating, that one won't require cert revocation.

I changed my Google, Facebook, Evernote and Amazon passwords. I also set up double identification for Evernote.

Are accounts using two step verification safe? Or at least safer?

Malor wrote:

Remember that you should revoke and reissue certificates that were used on any OpenSSL-enabled site. Otherwise, bad guys (including the government) may be able to impersonate your site, because they could have invisibly grabbed the server's private key without any visible log or evidence of intrusion. And they will definitely be able to surveil all traffic on your site until you generate a new key.

I like Malor's point, but I don't just think governments are the concern. Anyone who deals in the stealing of user information probably setup scans for this the moment it was made public - I know I would have. An undetectable method to grab login details, credit card details, as well as private key information? It'd be like the best holiday ever. Imagine if you knew about this before it went public...

So yeah, it sucks having to change passwords (I'm not a fan, really), but it's something you should take seriously. You should also make sure you monitor your credit cards that you use online, because you have no guarantee.

Transitive trust is typically not a good security model. The fundamental problem is that you have no easy way of knowing that the people you trust are as careful as you are, and vice-versa. And you have no way to know if someone is applying rubber-hose persuasion to them, either.

It might be a little better than what we have, but it's not actually *good* in any real sense.

This vulnerability extends to the servers that run the Canadian Government's Tax servers. This is unfortunate, as our taxes are due at the end of the month.

http://www.openbsd.org/cgi-bin/cvswe...

edit: OpenSSL as a tragedy of the commons - so many people using it, no one putting resources into making it better.

http://www.washingtonpost.com/busine...

Theo pointed out that OpenSSL explicitly disabled OpenBSD's ability to catch this bug. Because, apparently, some system mallocs were slow some of the time, they implemented their own, bypassing the system's completely, so all the effort the OpenBSD developers put into catching bugs of this type was wasted, in this case.

There's probably a whole bunch more bugs in OpenSSL that their lousy internal malloc has hidden.

dejanzie wrote:

Are accounts using two step verification safe? Or at least safer?

Two-factor is an interesting question. I'd say definitely safer, but definitely not completely safe. Two-factor works by establishing a shared secret between your phone and the server, so that you can both perform the same hashing operation and get the same answer. To perform that validation on login, the server needs to have that shared secret decrypted in RAM somewhere, so if the code doing that is in the same process as the web server, there's a chance an attacker could use Heartbleed to expose that secret.

The secret is just a random number, though, so actually identifying one, and linking it to a specific account so that it can actually be used to gain access, is non-trivial (not impossible, just difficult). If you're feeling particularly paranoid, I'd go ahead and re-setup two-factor, so you get a new shared secret.

FWIW, SMS-based two-factor generally works the same way, it's just that the secret is kept only on the server; it's used to generate the code that's sent to you, and then later used again to validate it. It'd be just as vulnerable to scraping the shared secret as a mobile app-based solution.

http://www.tedunangst.com/flak/post/analysis-of-openssl-freelist-reuse

Interesting analysis of the bug: it was only undetected for so long because OpenSSL used a custom memory allocator, instead of the default system allocator which would most likely just have crashed instead. An issue that was reported as a problem to the project 4 years ago.

For something security critical, would you not use the most stringent, paranoid memory access checking you can possibly get?

Yeah, this is definitely get worse. And worser. And worsest.

CloudFlare is claiming they've tried to use Heartbleed to pull private SSL keys and they can't.

I'll let you guys get into the nitty gritty there and report back.

XKCD explains the basics of how it works, if even slightly accurate then that is a pretty wtf /embarrassingly bad oversight.

'course hindsight and all but come on now...not one person working on the system bothered to test with string length greater than the requested response?

NSA Said to Exploit Heartbleed Bug for Intelligence for Years via Bloomberg

The U.S. National Security Agency knew for at least two years about a flaw in the way that many websites send sensitive information, now dubbed the Heartbleed bug, and regularly used it to gather critical intelligence, two people familiar with the matter said.

The NSA’s decision to keep the bug secret in pursuit of national security interests threatens to renew the rancorous debate over the role of the government’s top computer experts.

Heartbleed appears to be one of the biggest glitches in the Internet’s history, a flaw in the basic security of as many as two-thirds of the world’s websites. Its discovery and the creation of a fix by researchers five days ago prompted consumers to change their passwords, the Canadian government to suspend electronic tax filing and computer companies including Cisco Systems Inc. to Juniper Networks Inc. to provide patches for their systems.

...

trueheart78 wrote:

NSA Said to Exploit Heartbleed Bug for Intelligence for Years via Bloomberg

The U.S. National Security Agency knew for at least two years about a flaw in the way that many websites send sensitive information, now dubbed the Heartbleed bug, and regularly used it to gather critical intelligence, two people familiar with the matter said.

And there's the other shoe.

IMAGE(https://pbs.twimg.com/media/Bk-NmSBIgAEtEPL.jpg:large)

Boy, you gotta love this quote from that article about the NSA:

“This administration takes seriously its responsibility to help maintain an open, interoperable, secure and reliable Internet,” Shawn Turner, director of public affairs for the office, said in the statement. “Unless there is a clear national security or law enforcement need, this process is biased toward responsibly disclosing such vulnerabilities.”

In other words, they reveal giant flaws in the Internet if they feel like it, if it doesn't unduly damage their ability to spy.

Also, it's worth pointing out that the NSA knew about this bug right away, meaning that the chance that it was deliberately planted is much, much higher than it was.

Planting the bug on New Year's Eve is one of the better ways to be sure nobody will look at the code.

garion333 wrote:

CloudFlare is claiming they've tried to use Heartbleed to pull private SSL keys and they can't.

I'll let you guys get into the nitty gritty there and report back.

Aaaaaand, hacked. So, Cloudflare's challenge to use Heartbleed to steal data has worked and they've retracted their comments that it's tough to impossible to use.

Obviously, since NSA has said they've used it then we know it was used, but now Cloudflare has backtracked and said they were wrong.

garion333 wrote:
garion333 wrote:

CloudFlare is claiming they've tried to use Heartbleed to pull private SSL keys and they can't.

I'll let you guys get into the nitty gritty there and report back.

Aaaaaand, hacked. So, Cloudflare's challenge to use Heartbleed to steal data has worked and they've retracted their comments that it's tough to impossible to use.

Obviously, since NSA has said they've used it then we know it was used, but now Cloudflare has backtracked and said they were wrong.

Story on The Verge

Heartbleed Challenge Results