An Untrustworthy TLS Certificate in Browsers

The major browsers natively trust a whole bunch of certificate authorities, and some of them are really sketchy:

Google’s Chrome, Apple’s Safari, nonprofit Firefox and others allow the company, TrustCor Systems, to act as what’s known as a root certificate authority, a powerful spot in the internet’s infrastructure that guarantees websites are not fake, guiding users to them seamlessly.

The company’s Panamanian registration records show that it has the identical slate of officers, agents and partners as a spyware maker identified this year as an affiliate of Arizona-based Packet Forensics, which public contracting records and company documents show has sold communication interception services to U.S. government agencies for more than a decade.

[…]

In the earlier spyware matter, researchers Joel Reardon of the University of Calgary and Serge Egelman of the University of California at Berkeley found that a Panamanian company, Measurement Systems, had been paying developers to include code in a variety of innocuous apps to record and transmit users’ phone numbers, email addresses and exact locations. They estimated that those apps were downloaded more than 60 million times, including 10 million downloads of Muslim prayer apps.

Measurement Systems’ website was registered by Vostrom Holdings, according to historic domain name records. Vostrom filed papers in 2007 to do business as Packet Forensics, according to Virginia state records. Measurement Systems was registered in Virginia by Saulino, according to another state filing.

More details by Reardon.

Cory Doctorow does a great job explaining the context and the general security issues.

EDITED TO ADD (11/10): Slashdot thread.

Tags: , , ,

Posted on November 10, 2022 at 9:18 AM • 29 Comments

Comments

Clive Robinson • November 10, 2022 9:52 AM

@ Bruce, ALL,

Re : CA’s

“and some of them are really sketchy:”

Only some?

Most western nations like America, Australia… etc have legislation “to compell” in one way or abother.

Others have placed staff in CA’s or by financial manipulation (RSA) have gained sympathetic help.

But mostly, due to the cut-throat nature of the business most CA’s have cut back not just security staff but security systems, thus they are in effectct a “push over” even for “script-kiddy” like attackers.

The real cause of the problem and why it’s so easy to get security credentials that are at best highly questionable is as I mention from time to time,

“Hierarchical Trust Systems”

In fact any and all Hierarchical poeer structures are by definition “corrupt”. Due to the way power gets vested in the very top of the pyramid and every human has “a price”, every system “a vulnerability”…

Yet we do not do research to get rid of hierarchies,

“Why?”

Are we complicitly corrupt?

Clive Robinson • November 10, 2022 10:42 AM

@ Bruce, ALL,

Re : The dificulties.

“Cory Doctorow does a great job explaining the context and the general security issues.”

Long story short, all information security is built out of a single foundation some call it a “Shared Secret” others call it a “Root of Trust”.

The primary requirment of a “Root of Trust” is not actually that it remains a “secret” but that it remains “unguessable” thus “unknown”. To see the distiction all Public Key Certificates contain as a minimum two “Roots of Trust” as Primary numbers that are multiplied together. It’s the “multiplication in a prime field” that –hopefully– keeps either Prime from becoming “known” to an attacker.

Thus the question of how to find “unguessable” numbers falls to what we chose to call “True Random Generators”(TRNG / TRBG) of numbers or individual bits[1].

You could call TRNG’s the fundemental problem of “information security” as that is effectively what they are.

This has been discused a few times over the years on this blog, one such was nearly a decade ago when similar PKcert issues arose,

https://www.schneier.com/blog/archives/2013/09/new_nsa_leak_sh.html/#comment-204119

From the discussion you can find earlier refrences to earlier discussions on TRNGs on this blog as they go back a near decade before that.

As far as I’m aware this was the first place on the Internet where open conversations on TRNGs their strengths, weaknesses and darn right failings happened so freely with so much exchange of information.

[1] To a computer all numbers are “integers” that may or may not have some structure. Such structure is generally described in “Abstract Data Types”, the base of which is a “bit” that then get aggregated into larger sets known informally as “bag of bits”.

Peter Pearson • November 10, 2022 12:55 PM

So, can we finally get some attention on the need for users to vet their browsers’ root certificate databases? I understand that Google and Mozilla need to include the Elbonian Post Office’s root certificate in order to provide a smooth experience for their Elbonian users, but to the vast majority of users, that certificate is much more a threat than a blessing.

- • November 10, 2022 2:57 PM

@Peter Pearson:

“So, can we finally get some attention on the need for users to vet their browsers’ root certificate databases?”

Maybe we should first get the browser developers to make it easy for ordinary users/people to do so, instead of needing a Doctorate in meta-gymnastics.

You would almost think that Apple’s Safari, Firefox, and Google’s Chrome developers have very much gone out of there way to make what should be a simple task difficult.

Makes you wonder why they would do that, and who gains by it, certainly not the ordinary users/people of their products.

JonKnowsNothing • November 10, 2022 3:35 PM

@All

I have often wondered why my browser or system has to be pre-loaded with bloatware designed for countries I will never visit, the languages I don’t read, write or understand or any of a number of “localization” schemes that are useful only in the certain locales. Same with downloadable fonts and an entire list of stuff that is context specific.

It’s appeared to be a “deliberately stupid” scheme to expose a device to ez-pz compromise. (1)

If you “follow the money” version of dots there is only one path…

===

1) One excuse is that the devices are mfg in one place and distributed globally. So they have to include every possible configuration but they leave all the leftovers afterward. Just in case you take a crash course in Etruscan or want to flip over to Babylonian.

lurker • November 10, 2022 8:59 PM

@- • , a Doctorate in meta-gymnastics.

Nope, just a bit of experience sifting through gizzard contents. So I find my browser’s Cert. store:
Q1, what’s XYZ doing in there?;
Q2, what will break if I remove it?

Ted • November 10, 2022 10:20 PM

I’d almost like to have a diagram showing all the connections in the article. There’s something intriguing under every rock and around every corner.

Is Mozilla the only one with questions? Apple, any thoughts? I see Apple appears to have three TrustCor certs.

“Available trusted root certificates for Apple operating systems”
https://support.apple.com/en-us/HT209143

Surely this won’t be the last we hear about this… all of it… is so spooky.

“The researchers speculated that the system is only used against high-value targets within short windows of time.”

I may inadvertently start checking out website CA’s just out of sheer curiosity.

Gert-Jan • November 11, 2022 6:05 AM

I’m running a firewall on my computer. Whenever a “new” application tries to access the internet, the firewall prompts me to allow or deny this.

It would be a nice security improvement (setting) if web browsers did the same thing for root certificates.

When I go to my banking website, which is normally vouched for by DigiCert Global Root CA, and all of a sudden my browser would ask me if it is okay to trust GUANG DONG CERTIFICATE AUTHORITY CO.,LTD., I would think twice before I would clik the Trust button…

In other words, why not make the list of root certificate providers and opt-in feature?

This wouldn’t solve all the problems, but it would reduce the attack surface substantially.

Rowena • November 11, 2022 9:26 AM

@ lurker,

Nope, just a bit of experience sifting through gizzard contents. So I find my browser’s Cert. store:
Q1, what’s XYZ doing in there?;
Q2, what will break if I remove it?

In my experience, it’s more like “here’s a list of 200 authorities, shown 6 at a time, and how the hell do I remove them without like 600 separate clicks?” “Ah, Firefox supports multi-select”, one might think. Nope; if you select the whole range, you’ll have selected folders, and that disables the Edit Trust button. Those who meticulously de-select the folders may be surprised to find that the Edit Trust button then opens about 150 separate dialogs, one after another. It’s moot anyway, ’cause one can’t see the trust settings on the Authorities list, and stuff will sneak into there at every (likely automatic) update.

The “real” way to disable the built-in authorities in Firefox or Chromium, for those wondering, is to make sure the browser can’t load the actual “nssckbi” library. On an ELF platform, replace it with a real and empty ELF library; using the “bwrap” tool, a.k.a. bubblewrap, is the easiest way, ’cause some browsers can get past $LD_PRELOAD unless you hook the dlopen() function. You can then add your own authorities—or individual certificate exceptions, which I recommend for sensitive contexts such as banking profiles. (You’ll find, by the way, that Chromium tries to validate something signed by “Google Trust Services” several times before you load any page, even if you think you’ve blocked all Google telemetry etc. and configured URLAllowlist and URLBlocklist very strictly.)

lurker • November 11, 2022 2:32 PM

@Rowena

Of course finding the Cert.Store is only half the fun. Like the Cookie Store, and Img.Cache, these days browsers wrap it up in a SQLite.db to discourage sticky fingers. FF describes some as Builtin Object Token and some as Software Security Device??

And why do some confidently set an expiry date 30 years out? while gmail are currently rolling over a new cert every 14 days?

Rowena • November 11, 2022 8:18 PM

lurker, I’m not too familiar with the format of the certificate store, but I know “certutil” is the command-line tool to work with it. It is… not user-friendly, especially when trying to edit an existing database. But if you save a bunch of TLS server certificates (in Firefox, click the lock, “connection secure”, “more information”, then “PEM (cert)” beside “Download:”), there’s a tool on Github to generate a cert_override.txt that can be dropped into a Firefox profile directory. Or for Chromium, run for each certificate:

cat foo.pem | openssl x509 -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64

Then prepend “–ignore-certificate-errors-spki-list=” to each hash and add these as command-line options.

It’s easy enough to script other of these to auto-generate, on browser startup, from a PEM directory stored in git. No SQL hackery required, though perhaps one could do similarly for a bunch of CA certificates and use certutil to re-create an NSS database from scratch each time (as I imply above, it’s easier to delete the old one than try to fix it). Probably a libnssckbi.so could be generated too; or to be a bit more clever, use a DLL constructor function to populate the list on dlopen. Debian-derived distributions keep a bunch of PEM certificates at /etc/ssl/certs that could be used for this purpose (and will ask before adding CAs there, if configured to do that); OpenSSL uses it but for some reason the packaged Firefox and Chromium don’t.

Rowena • November 12, 2022 10:12 AM

keiner, I don’t know why those instructions didn’t work. Based on the state of the user interface, I suspect it’s not a part of Firefox that gets a lot of attention from programmers and testers. Deleting the profile’s security database, though, is the exact wrong thing to do, because that’s where the overrides of the default trust settings would be stored. Any certificate that’s built in (to nssckbi) and not seen in the profile will simply be re-imported and re-trusted at the next startup. Therefore, you also can’t expect a “deleted”/distrusted certificate to disappear from your list. It’ll be there forever, but when you click “Edit Trust”, the checkboxes should be clear, indicating it’s not trusted for any purpose.

The certutil manual page says cert8.db, key3.db, and secmod.db are “legacy” databases, while cert9.db, key4.db, and pkcs11.txt are their SQLite replacements (yes, it says that about the .txt file too). It’s odd that you have both present. You could try deleting the legacy set only.

Denton Scratch • November 13, 2022 6:28 AM

Every country that has electronic ID also needs its own CA to sign that ID. And every country whose autocratic leader wants to appeal to nationalism has to have its own CA. And then there are all the countries that (for justifiable reasons) want to offer their citizens certs that aren’t controlled overseas.

For example, I don’t need any root certificates that are issued in Turkey; the Turkish government has form for mishandling certificates. I don’t need EV certificates at all; hardly anyone serves them any more.

I wish these “certificate managers” gave you a link to good information about why each certificate is included in the browser. If that information is presented only in Hungarian, for example, I can delete it. But even if you google the CA, you have to drill down to learn that root certificate X is actually issued by some low-rent Japanese TV manufacturer, or whatever.

Most people could manage most of the time with nothing but the LetsEncrypt roots. And I wish they’d loosen-up on self-signed certificates; I never understood why the browsers got so arsey about them. They could just pop up a warning; something like “This site is protected by a self-signed certificate; proceed at your own risk [more info]”. And then TOFU.

Perhaps national CAs could be part of a locale bundle; so if you install the Finnish locale, then and only then do you get the Finnish national CA’s roots.

And I’d like to seee the browsers immediately evict from their root bundles any CA that has the slightest whiff of scandal hanging over it. They are terribly slow to evict even CAs that have been convicted of crimes, or have been hacked. Then a couple of years later, they reappear with no prominent announcement. Of course, the browsers run certification programmes for CAs; you have to pay to be certified, so there’s a profit to be made there for the browser vendors (and the CAs who pay would also prefer that users not be able to easily edit their CA lists).

A neat feature would be the ability to import a “safe” CA list, published by e.g. the EFF.

It’s too bad that CACert got ejected; they had better processes than the commercial vendors, but their privacy processes and their use of Web Of Trust were incompatible with the browsers’ CA certification rules (i.e. the rules were wrong).

Clive Robinson • November 13, 2022 8:02 AM

@ Denton Scratch, ALL,

Re : Wrong rules who’s at fault.

You note of CACert being ejected,

“incompatible with the browsers’ CA certification rules (i.e. the rules were wrong).”

So the questions that naturally arise are who set the rules? And why they set them the way they did?

In the past I’ve tried finding out those answers, and it appears to be that “money talks” is the primary answer, most notable of the visable suppliers of which being Google…

So as you also note,

“the CAs who pay would also prefer that users not be able to easily edit their CA lists”

Would suggest one reason, why browser cert DB’s are near impossible for ordinary users to manage. But I suspect there are other more shall we say “compelling” reasons at play such is the nature of politics.

As @- said above,

“Maybe we should first get the browser developers to make it easy for ordinary users/people to do so, instead of needing a Doctorate in meta-gymnastics.”

Which most of the following comments mainly suggest would be a very good idea…

I do like your EFF idea, but I’m not sure they would with the way politics and US legislation works…

lurker • November 13, 2022 2:43 PM

@Denton Scratch, Keiner, All

re not wanting any certs issued in Elbonia, or only needing locale certs for Godwana,

Unfortunately it’s called the World Wide Web. eg. I found a couple of odd Chinese certs in my FireFox, and thought they might have been added during my travels in those parts of the internet, but FF never asked me. It’s been a few years now since I last used Safari, but it always used to pop up a dialog for unknown certs (iirc):

Site XYZ wants to use an untrusted cerificate.
[View Certificate]
Do you wish to use this certificate?
[Deny] [Use this Once] [Use permanently]

And when necessary it used the same dialog replacing “untrusted” with “self-signed”.

Firefox is not my daily browser, but it does allow in Settings > Privacy & Security > Certificates for the user to not trust individual certificates.

So I did a slash and burn exercise:
disconnect internet
rm -R ~/.mozilla ~/.cache/mozilla /tmp/mozilla_username
sudo rm -R /etc/ca-certtificates/mozilla
verify Firefox has no stored certificates
restart

Firefox had repopulated itself with 135 items, Builtin Object Tokens and Software Security Devices. So it seems you can use forks and soap[1] to adjust your personal trust in bad certificates, but you can never remove them.

[1] more metagymnastics: https://en.wikipedia.org/wiki/The_Hunting_of_the_Snark

Rowena • November 13, 2022 3:16 PM

I posted a reply to keiner yesterday, and it’s still held up in moderation. With regards to lurker’s message also, “deleting” a few certificate authorities is really not what you want to do—nor what the “delete” button actually does. You actually want to store non-default trust settings for some CAs, or remove all the CAs.

Any certificates not in your Firefox user profile, but found in nssckbi, will be re-added to the profile at startup, with default trust settings. So you either knock out nssckbi entirely, or you deal with them being forever in your list—but not necessarily trusted. Click on whichever one you tried to delete and look at the checkboxes; if they’re clear, the operation worked.

(As noted previously, though, there’s no way to see a cert’s trust settings without clicking on it, and there are like a hundred and fifty to check. Knocking out nssckbi and starting from scratch may seem crazy, but it’s really the more reasonable and maintainable long-term option. Especially if you’re already maintaining a list in /etc/ssl and can hack up a simple shell script.)

keiner • November 14, 2022 3:26 AM

@Rowena: Thanks for explaining, then the text in the Firefox Settings (“Delete or distrust…”) is at least ambigous (or may I say deceptive?).

I removed trust from the three Trustcor certs for the start…

GG • November 15, 2022 5:47 AM

Has any abusive cert be signed by them be seen in the wild?

It may be that the country or company is untrustworthy, but unless they actually break the CAB Forum rules, they can be trusted, right?

Isn’t that the whole point of the CAB Forum?

Rowena • November 15, 2022 11:29 AM

@ GG,

Has any abusive cert be signed by them be seen in the wild?

No, at least not by anyone who recognized it and was willing and able to report it. If there were proof, it would certainly have been mentioned in the bug report and stories.

It may be that the country or company is untrustworthy, but unless they actually break the CAB Forum rules, they can be trusted, right?

Anyone “can be” trusted, and this certificate is trusted, in the sense that people rely on them for security. In principle, they can simultaneously be untrustworthy.

Isn’t that the whole point of the CAB Forum?

Yes, the CAB Forum does determine which certificates are “trusted” in the sense of “trusted systems“. That’s mostly because, back in the 1990s, nobody had a more practical idea. Given the semi-illegal status of strong cryptography, and the niche status of SSL, browser makers couldn’t really force all the domain registries and registrars to handle it, for example. And S/MIME email cryptography would have been even more impractical if we’d had to convince every email provider to support it.

C S • November 16, 2022 5:44 PM

Burn it all to the ground. The basic foundation of SSL is broken, and we need to reboot the trust model completely.

This would not mean dumping the technical parts of the certs thankfully, and a workable solution could probably be tacked together with yet ANOTHER txt record in your (Hopefully now signed) DNS records.

But much like a Allowed senders list for your mail domain, you should be able to designate the CA’s your organization actually uses, and automatically reject any CA issuing certs not on that list.

Otherwise the idea of SSL is logically broken.

This isn’t new, people here and other places have literally been talking about this for decades. What hasn’t happend is action. Instead of putting a stake in the heart of the problem, Google and others started stapling individual certs internally to their apps and libraries. It staved of some active attacks, but at the cost of leaving the entire internet permanently vulnerable to forged certs from technically valid CA’s.

Clearly, based on history, many of these CA’s should trigger a warning prompt, but it would be better if the code that runs the internet was smart enough to let he Saudi or North Korean CA sign code or certs in it’s scope of influence and warn about certs signed by their CA in other scopes. Also to autoreject certs issued by a valid CA for scopes that are declaring a specific CA.

While I hate to lean on DNS as the glue for this, people are familiar with it, and it will make it hard to permanently lock an organization to an old CA as long as they still control their domain registration, which generally does not require them to have a working SSL/CA relationship to clean up.

Rowena • November 16, 2022 10:16 PM

@ C S,

a workable solution could probably be tacked together with yet ANOTHER txt record in your (Hopefully now signed) DNS records.

But much like a Allowed senders list for your mail domain, you should be able to designate the CA’s your organization actually uses, and automatically reject any CA issuing certs not on that list.

You’re pretty much describing DNS Certification Authority Authorization, and the CA/Browser Forum has mandated it for all certificate authorities (in most circumstances) since 2017.

I see no CA/B requirement for anyone to validate DNSSEC, though. A CA is allowed to treat a lookup failure as permission to issue if, among other things, there’s no “DNSSEC validation chain to the ICANN root”. But the RFC specifically says DNSSEC is “not required”, and in the absence of lookup failure, the CA/B rules say nothing on the matter. This is clearly ridiculous. Any CA relying on unsigned DNS data should be required to validate and store (for future audits) the DNSSEC opt-out(s) covering that data. Any CA relying on signed data should be required to validate and store the entire chain of such data.

Rowena • November 17, 2022 4:03 PM

Today in sketchy certificate authority news: e-Tegra, a Turkish CA, was using default credentials for some publically-accessible administration pages. Helpfully, the site told visitors which name and password to use. A different section of their website allowed anyone to create their own admin account—with no verification, of course—to look at things such as pictures of people’s passports.

The linked page references the “CA/B Forum Network Security Guidelines”. In fact, CA/B claim these to be requirements (not guidelines), despite several “SHOULD” statements and other escape clauses. Oddly, since this account was accessible outside a “Secure Zone”, the rules only required an 8-character password; whereas had access been properly restricted, it would’ve required 12. That seems backwards, although I think the accessibility outside the secure zone was, itself, a breach.

What’s the CA/B policy for handling a failure to meet requirements, particularly in such an egregious case? It seems like something that should be on their site; and given that it’s happened before, should be accompanied by some kind of history of additions, breaches, and removals. Maybe it’s buried in e-mail archives or meeting minutes, but I’m not seeing anything obvious.

lurker • November 17, 2022 5:56 PM

@Rowena, “What’s the CA/B policy for handling a failure?”

Does it matter what the policy is if no action is ever taken against violators? To see any change on the ground will need a few hands chopped off at noon on Friday in the town square (or a Europeanised equivalent). Otherwise it’s just more waffle, like the technical foundations of the internet built on “Requests For Comments”.

Rowena • November 17, 2022 6:24 PM

@ lurker,

Does it matter what the policy is if no action is ever taken against violators?

When I wrote “history of … breaches, and removals”, that was my intended subtext. I know stuff like this has happened before, and I recall some “too big to fail” events that made the CA/B look quite toothless. But I was expecting to find some pretense of consequences for failure; maybe some record of them having deleted a CA with no global significance, like Turktrust.

(I don’t recall exactly what happened with Turktrust, DuckDuckGo doesn’t find much, and they’ve got no Wikipedia page. I don’t see it in Firefox’s default list; but with no search feature, only a tiny box in which to view the list, and an unclear sort order, I can’t easily be sure they’re not present.)

Chris • November 23, 2022 4:18 PM

The best improvement I can think of is to have a set of CAs for each country. So a site with a url ending .uk could only be certified by a CA in the UK and subject to UK law. A site ending .ca would need a Canadian CA, etc.

Sites ending .us, .gov or .mil would need a CA in the USA. But .com would have to be a “free fire zone” for any CA. As would other top level domains not tied to any specific country.

This should make things safer for people who are not very security conscious (nearly everyone not a security expert).

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.