It’s always DNS

I host a few websites for myself and family on DigitalOcean.  Up until recently, I’ve always just spun up a new droplet for each site, so they were all fully independent from each other; this was the easiest and most convenient way to get a new site up and running without jeopardizing uptime on other sites if I made a mistake in configuration, and it was drop-dead easy to map a domain to a static IP.  It had some security benefits, too– if one site was compromised, it wouldn’t affect the rest.

But it was also maintenance-intensive.  I needed to login to multiple servers to run updates; adding plugins had to be redone over and over on each server; and obviously this was starting to get expensive.  So I decided to consolidate my multiple sites on one server, using a fancy feature of WordPress called… “Multisite“.  Imaginative name, I know.

The initial configuration went well, with no real hiccups (other than my accidentally rm’ing most of Apache’s configuration files– but a quick droplet rebuild took care of that1).  The trouble started when I had moved over the sites I was consolidating, and switched the domains to point at my new Multisite server.  I spent two hours trying to figure out why one of the domains refused to point at the new server, only to discover (drumroll, please)… it was DNS.  I use Pi-Hole on my home network to block malicious sites, but it also provides a DNS caching service which usually works great.  In this case, however, it was pointing me back at the old server over and over, until the TTL finally expired2.  A quick flush of the DNS cache, and I was able to see that the domain was correctly configured.  Fifteen minutes later, I had SSL up and my plugins configured.

So what’s the lesson in all this?  Even when you think it’s not DNS… it’s DNS.

  1. Yes, I could have restored the configuration without too much difficulty, but I was early enough in the build that it was faster to just start over.
  2. I did set the TTL to a very low number when I started this process, but the old value wasn’t updated until the original one expired.
Format Aside

A new reflection attack was unveiled today which can increase the size of a DDoS attack by 51,000-fold.  It uses memcached, an object caching system designed to speed up web applications, to amplify attacks against a target.  This represents a substantial increase from previous attacks, which have used network time servers to amplify attacks 58-fold and DNS servers to amplify attacks 50-fold.

Attacks seen this week have surpassed 500 Gbps, which is pretty amazing considering only a small percentage of publicly-available memcached servers are being used to launch those attacks.  It’ll be interesting to see if any larger attacks are launched in the coming weeks… and what their targets will be.

The article over at Ars Technica is pretty good, and is worth a read.

Format Aside

Another day, another vulnerability in a widely-used software package.  Today’s bug (dubbed Optionsbleed by Hanno Böck, the journalist who documented the vulnerability) can reveal passwords and other pieces of vital information to attackers.  While not as big of a threat as Heartbleed, a similar bug which allowed attackers to snag private encryption keys for servers (which is a Bad Thing, since this is how servers verify they are who they say they are; for an explanation of how this works, see my Asymmetric Encryption explanation from last year), this should still be regarded as a significant threat.

Patches are being rolled out now; patch your systems if you haven’t already.

Format Aside

The vulnerability was patched in WordPress v4.7.2 two weeks ago, but millions of sites haven’t yet updated.  This leaves them open to a vulnerability in the WordPress REST API, which can allow malicious actors to edit any post on a site.

Ars Technica has a very nice writeup on the effects of the exploit, which has resulted in the defacement of a staggering number of websites (including the websites of Glenn Beck, the Utah Office of Tourism, and even the official Suse Linux site).  Sucuri and Wordfence also have very good articles about the effects of the vulnerability.

If you have a WordPress site, you should immediately check to make sure you’re on the latest version (v4.7.2).

Format Aside

I’ve noticed a growing trend in more advanced computer users lately: some of them have begun advocating against using antivirus software.  Instead, they suggest using browser extensions like uBlock Origin (which I use and recommend), combined with safe browsing practices, to remove the need for antivirus software altogether.  Ars Technica did a very nice write-up on this trend today, and it’s worth a look.

For what it’s worth, I still use Avast as an antivirus package.  But it hasn’t alerted me to any issues or found any viruses in at least a year, so perhaps it’s time to consider freeing up some memory on my computer.

New Host!

I’ve finally moved to a VPS on DigitalOcean, from my previous (free) shared hosting.  I did this for a couple of reasons: first, while my hosting was free for a year with my domain name, that year was almost up.  To renew my hosting for the second+ year, I would have needed to pay $38.88/year; while that’s a decent price, I looked at my options and decided that moving to DigitalOcean wouldn’t cost much more (around $30 more across the year, since I use the weekly backups option), would give me much more control over my server (now I get SSH access!), and would centralize all of my VPS instances in the same place (I’ve used DigitalOcean for several years to host various projects).

Of course, as with so many things, this migration wasn’t sparked by a simple glance at the calendar.  While I’ve intended to move my host for the last month or two, the timing was decided by my messing up a WordPress upgrade on the old site at the beginning of December.  I used the automatic updater, ignored the warnings about making sure everything was backed up first1, and told it to apply the new version.  When WordPress exited maintenance mode, I was locked out of the administration dashboard.  The public part of the website was still up and running, but the backend was locked off.  Since I was entering finals week at my university, I decided to just let it be until I had some time to come back and fix it.  Worst-case, I had backups I could restore from, and I’d been meaning to migrate my site anyway.

Of course, things didn’t work out that way.  When I finally had some time on Christmas Eve, I discovered that a complete backup hadn’t been made in months.

Yes, I committed the cardinal sin of not verifying the state of my backups.  Apparently I’d screwed something up with their configuration, and I’d never tried to restore from them before and hadn’t noticed until I needed them.  At this point, I decided that if the backups weren’t working, there was no point in trying to recover on a host that I was going to be abandoning within a month, and I spun up a WordPress droplet on DigitalOcean to hold the rebuilt site.

I still had copies of all the content that was on the site, so I’d be able to restore everything without much trouble.  Some copy/pasting and time would be required, but I could get everything back to the way it was without too much trouble.  But before I did all of that, I thought “what if I’m overlooking something really simple with the old site?”  I did a little searching, and apparently W3 Total Cache, which I used to create static pages for my site and decrease load times, can cause problems with WordPress upgrades.  I disabled that via FTP2, reloaded the site, and I was able to access the admin area again.  Turns out the simple steps that you should take before completely rebuilding everything are actually worth it.

Since I had already spun up and started configuring my new site, I decided to press onwards.  My task was made considerably easier by my being able to access WP Clone on the original site, which let me move everything from my old site to the new one in just a few minutes.  I redirected the nameservers to DigitalOcean, and ran a few last checks before calling the bulk of my work done.

The next day, when I was tidying up some loose ends and preparing to get SSL set up, I realized that my email no longer worked– my email server resided on the same server that hosted my old website, which meant I needed to find a new solution.

While I have been meaning to setup my own email server sometime soon, I wasn’t confident in my ability to get it up and running quickly, and email is one of those vital services I depend on working 100% of the time.  In years past, I would have simply used Google Apps3 to host my email, but that is no longer the free option it once was.  Luckily, I found a solution thanks to Ian Macalinao at Simply Ian, which is to use Mailgun as a free email server.  Mailgun is designed to send out massive email blasts for major companies, but they also offer a free tier for people and companies that are sending out fewer than 10,000 emails per month.  I send out a fraction of that number, so this was perfect for me (and their mass email prices seem quite reasonable, so I might even use them for that if the need ever arises).  Ian handily provided a set of instructions for how to setup the proper routing, and, while some of the menu options have changed, I was able to get my new email up and running within a few minutes.

So I’d managed to get both the site and my email up and running, but I still couldn’t get SSL up and running.  For those that don’t know, SSL stands for Secure Sockets Layer, and it’s what powers the little green padlock that you see on your address bar when you visit your bank, or PayPal, or this website.  I wrote an explanation on how it works a while back, and I suggest checking that out if you want to learn more.
One of the benefits of hosting my website on a VPS is that I don’t need to use the major third-party SSL providers to get certificates saying my server is who it says it is; I can use the free and open Let’s Encrypt certificate authority instead.  Unfortunately, I just couldn’t get the certificate to work correctly; the automated tool was unable to connect to my server and verify it, which meant that the auto-renewal process wouldn’t complete.  I could have generated an offline certificate and used that, but the certificates only last ninety days and I wasn’t looking forward to going through the setup process every three months.4  I tried creating new Virtual Hosts files for Apache, my web server, but that just created more of a problem.  Eventually, I figured out that I had misconfigured something somewhere along the line.  Rather than try to figure out which of the dozens of edits I had made was the problem, I gave up and just reverted back to a snapshot I had made before starting down the rabbit hole.5  After reverting to back before my virtual hosts meddling, I was able to successfully run the Let’s Encrypt tool, generate my certificate, and secure my site.

Lesson learned!


Photo credit Torkild Retvedt.

  1. I didn’t actually ignore this warning.  I had a backup plugin configured on the site; I figured I could probably roll back if I really needed to.
  2. If you’re in a similar situation, just renaming the plugin folder to something else– w3-total-cache to w3-total-cache123, for example– will disable it
  3. Which is now G Suite, but that sounds silly.
  4. It’s a pretty straightforward and simple process, I just know that I would forget about it at some point, the certificate would expire, and the site would have issues.  If I can automate that issue away, I would much rather do that.
  5. Snapshots are essentially DigitalOcean’s version of creating disk images of your server.  I absolutely love snapshots; they’ve saved my bacon more than once, and I try to always take one before I embark on any major system changes.
Format Aside

Ars Technica did a nice job of creating an impartial write-up on why Hillary Clinton used an external email server, and how it was actually used.  It sounds to me like there’s an institutional history of using private email to conduct business, largely due to obstructive or incompetent IT services (in fairness to the State Department IT team, there are likely a number of complicated policies and legal requirements that they’re trying to work around, which is difficult).  Still, that’s not an excuse to use a home server to manage official communication– if you must use your own email address, at least use something like Google Apps or Microsoft Exchange Online, where you have teams of people professionally managing the email environment 1.

It’s also interesting to see that the NSA basically shot down any possibility of her getting a secured mobile device; I would have thought that providing the Secretary of State– the person who comes fourth in the presidential line of succession– with secure communications at all time would be a priority for them.

You can read the full story here.


  1. Of course, there is still the issue of all email traffic being unsecured and transmitted in plaintext.  But you could use a PGP solution to reduce risks there.

Hacking the Hackers

Have you ever heard of Hacking Team?  It’s an Italian company specializing in “digital infiltration” products for governments, law enforcement agencies, and large corporations.  Simply put, they sell hacking tools.

You might think, given their business model, that they would monitor their own security religiously.  Last year, however, they were hacked.  Majorly hacked.  “Hundreds of Gb” of their internal files, emails, documents, and source code for their products were released online for all to inspect, as were their unencrypted passwords 1.  Also released was a list of their customers, which included the governments of the United States, Russia, and Sudan—the last being a country controlled by an oppressive regime that has been embargoed by the E.U. 2

Last Friday, the person claiming responsibility for the attack, “Phineas Phisher”, came forward with details about how they did it.  It’s worth reading through if you’re interested in security; if you’d like an explanation geared more towards the layperson, Ars Technica has a pretty good write-up/summary of the attack.

I was particularly struck by how they gained access to the network.  According to Phineas,

Hacking Team had very little exposed to the internet. For example, unlike Gamma Group, their customer support site needed a client certificate to connect. What they had was their main website (a Joomla blog in which Joomscan didn’t find anything serious), a mail server, a couple routers, two VPN appliances, and a spam filtering appliance… I had three options: look for a 0day in Joomla, look for a 0day in postfix, or look for a 0day in one of the embedded devices. A 0day in an embedded device seemed like the easiest option, and after two weeks of work reverse engineering, I got a remote root exploit…  I did a lot of work and testing before using the exploit against Hacking Team. I wrote a backdoored firmware, and compiled various post-exploitation tools for the embedded device.

Basically, to avoid detection, Phineas discovered a unique vulnerability 3 in one of their embedded devices (likely one of their routers), figured out how to use it to get into the rest of the network using that vulnerability, and then carried out the attack through that piece of hardware without anybody noticing.  No matter your feelings about the attack, this is an impressive feat.


 

  1. By the way, here’s some advice: if you are in security (or anything, really, this isn’t security-specific) you should really make sure your passwords are more secure than “P4ssword”, “wolverine”, and “universo”.  Use a passphrase instead.
  2. As an Italian company, this means that they were technically violating the embargo.
  3. These unique vulnerabilities are called a “zero-day” in computer security circles, because the hackers find it before the company maintaining the software or device does— so once the company finds it, they have zero days to mitigate damage.

What is asymmetric cryptography?

Whitfield Diffie and Martin Hellman were jointly awarded the 2015 ACM A.M. Turing Award today.  Their 1976 paper, New Directions in Cryptography, essentially created asymmetric cryptography.  Today, asymmetric cryptography secures our online communications—from PGP-secured texts, emails, and files, to TLS and SSL-secured websites (including this one).  So how does asymmetric cryptography work, and how is the Diffie-Hellman key exchange more secure than older methods of encryption?

Symmetric encryption

Symmetric encryption relies on a key 1 shared between two or more people.  A message is encrypted using this key, and can then be decrypted by the same key held by somebody else.  Think of it like the front door of a house.  Alice has a key to the door, so she can lock and unlock the door.  Bob also has a key, so he can also lock and unlock the door.  In fact, anyone with a copy of that key can both lock and unlock the door whenever they want.  In the case of a message, this means that anyone with the right key can encrypt (lock) the message, or decrypt (unlock) the message.

It’s possible to break symmetric encryption 2, though it takes time.  Perhaps one of the most famous examples is from World War II, when the Allies were struggling to crack encrypted Nazi communications.  The encryption was created with a key that changed daily, and through the use of the Enigma machines.   The cryptography was eventually broken, but largely through the skill of the codebreakers, poor operating practice from some of the German operators, and the capture of key tables and hardware by the Allies.

Asymmetric encryption

Asymmetric encryption, in contrast to symmetric encryption, uses a pair of keys to encrypt messages.  One of the two keys is made public to everyone, and one is kept private (the two types of keys were called, cleverly enough, the public key and the private key, respectively).  Messages encrypted with the public key can only be decrypted using the private key 3, which ensures that the contents of the message can’t be read by anyone except the holder of the (hopefully secure) private key.  So if Alice wants to send an encrypted message to Bob, she starts by finding his public key.  She then encrypts her message using that, and sends it to Bob.  When Bob receives it, he uses his private key to decrypt the message.  If he wants to respond, he can encrypt his reply using Alice’s public key, and the cycle continues.  Since the public keys are usually published or exchanged in a way that lets each party be confident that it belongs to whomever they are sending it to, this ensures that the identity of the recipient can be verified.  Alice knows that only Bob can unlock her message, and Bob knows that only Alice can unlock his.

This is commonly used on websites that are secured by SSL/TLS (including this one).  Pretty much every major website is secured via SSL, and browsers will display a green padlock in the address bar of secured sites.  This serves two purposes; it will prove that the website belongs to whomever it purports to belong to, and it encrypts traffic between your computer and the website so that it can’t be read by attackers, your ISP, or others who may have a vested interest in what you do.
This works in exactly the same way that the messages between Alice and Bob did.  When you visit a website secured with SSL, your browser and the server exchange public keys.  The server encrypts traffic to you using your public key, which your browser decrypts.  And your browser encrypts traffic to the server using the server’s public key, which the server decrypts.  Anyone trying to listen in on the conversation your browser and the server are having will hear nothing but random gibberish.  There’s one additional thing that your browser does to ensure that it’s not talking to a fake server that’s pretending to be the real website: it takes the public key presented by the server, and it compares it to a repository of public keys.  If it matches, it’s the real server.  If it doesn’t, it could be an imposter– and somebody could be listening in.

So the next time you’re wandering around the web, take a minute to appreciate that little green padlock in the corner of your screen, and think about the incredibly complicated math that underpins security on the internet.  It works invisibly to keep your communications safe, secure, and most importantly—private.


 

I’m not a cryptographer or a security specialist, just somebody who enjoys reading and learning about security.  If you think I left out something important, please send me an email so I can fix it.

  1. Essentially, a key is a piece of really complicated math.
  2. It’s also possible to break asymmetric encryption, or any encryption
  3. Basically, the message is sent through a mathematical formula that only works one way… unless you have the incredibly complicated and unique formula that comprises the private key.