Tag Archive for: Maintenance

Removing support for TLS 1.0 and TLS 1.1

TL;DR

For security reasons, it is best practice to disable TLS 1.0 and TLS 1.1, but before you do this you may need to weigh up the risks to traffic from old browsers.

After disabling TLS 1.0 and TLS 1.1 any visitors using old browsers won’t be able to access your site.  If you are accepting credit card payments through your website then your customers security is more important but if you have a public information site then this may not be the case.

Don’t I always want the best security?

Please don’t get us wrong. We are NOT advocating blindly reducing security. This post is very much a response to customers that come to us wanting changes that will break their sites in order to get a perfect score or tick a compliance box. We can usually come up with a best of both worlds once we show the exact implications of the change.

What’s the fuss about?

GlobalSign’s What’s Behind the Change? paragraph sums this up nicely:

Various vulnerabilities over the past few years (e.g., BEAST, POODLE, DROWN…we love a good acronym, don’t we?) have had industry experts recommending disabling all versions of SSL and TLS 1.0 for a while now. PCI Compliance was another driving factor. On June 30, 2018, the PCI Data Security Standard (DSS) required that all websites needed to be on TLS 1.1 or higher in order to comply.

The RFC 7525 from 2015 stipulates that implementations should not use TLS 1.0 or TLS 1.1:

   o  Implementations SHOULD NOT negotiate TLS version 1.0 [RFC2246];
      the only exception is when no higher version is available in the
      negotiation.
      Rationale: TLS 1.0 (published in 1999) does not support many
      modern, strong cipher suites.  In addition, TLS 1.0 lacks a per-
      record Initialization Vector (IV) for CBC-based cipher suites and
      does not warn against common padding errors.
   o  Implementations SHOULD NOT negotiate TLS version 1.1 [RFC4346];
      the only exception is when no higher version is available in the
      negotiation.
      Rationale: TLS 1.1 (published in 2006) is a security improvement
      over TLS 1.0 but still does not support certain stronger cipher
      suites.

Qualys SSL Labs have reduced their grading for servers which support TLS 1.0 or TLS 1.1

Assessing the risk

Who won’t be able to access my website if I disable TLS 1.0 or TLS 1.1? Generally speaking browsers before 2013 will have trouble.  Most popular clients affected are old Android phones and old versions of Windows with Internet Explorer 10.  For the exact Android versions and other affected clients this is a nice breakdown.  As you’d expect the number of visitors with these old clients will vary according to your user base.  It’s best you check your site’s analytics to inform your decision.

Again, you can take into account how important encryption is for your website.  For example, at the time of writing it’s interesting to note that paypal.com has removed support for TLS 1.0 & 1.1 whilst google.com has not.

Summary

So what does this mean?  Lets give some examples…

If security is important to you; perhaps you have an e-commerce site taking payments or you are a IT consultancy like ourselves where people wish to share private information. You must disable old SSL/TLS protocols so that the only way people can communicate with your site is as secure as possible.

If accessibility is important to you; perhaps you are trying to share public information, be it a marketing or public resources site. It maybe worth supporting old protocols to allow your message to be shared as wide as possible.

Remember; it maybe typically called a sales “funnel” but traffic doesn’t have to end up in just one place. Users not supporting the right levels of security can be redirected to alternative pages where they can be contacted in other ways.  Why lose a sale when you don’t have to!

 

We’ve intentionally painted with broad strokes in this blog post.  We’re happy to give specific advice if you contact us and feel free to leave a comment 🙂

Warning sign - Outages

Common warning signs before server outages

Everyone knows that server outages and server down time cost. It directly affects your business in a number of ways including:

  • Loss of opportunities
  • Damage to your brand
  • Data loss
  • Lost sales
  • Lost trust

It’s essential to stay on top and ahead of any potential downtime.

Here are three areas where you need to be ahead of the curve:

Know your limits / server resources

Physical resource shortages

A common cause of downtime is from running out of server resources.

Whether it is RAM, CPU, disk space or other, when you run out, you risk data corruption, programs crashing and severe slowdowns to say the least. It is essential to perform regular server monitoring of your resources.

One of the most important; yet overlooked metrics, is disk space. Running out of disk space is one of the most preventable issues facing IT systems in our opinion.

When you run out of disk space, your system can no longer save files, losing data and leading to data corruption.

Often your website might still look like it is up and running and it’s only when a customer interacts with it, perhaps uploading new data or adding an item to a shopping basket, that you find it then fails to work.

We see this happen most frequently, when there is a “run-away log file” that keeps expanding until everything stops on the server!

CMS systems like Magento fall particularly prey to this as they often have unchecked application logs.

Internally, we record all server resource metrics every 10 seconds onto our MINDER stack and alerts will be raised well in advance of disk space running out. You don’t need to be this ‘advanced’ – you could simply have a script check current disk space hourly and email you if it is running out.

Configured resource shortages

Another common resource limit is a misconfigured server.

You could have a huge server with more CPU cores, RAM and storage than you could dream of using, but if your software isn’t configured to use it it won’t matter.

For example, if you were using PHP-FPM, and hadn’t configured it correctly, it would only have five processes running to process PHP. This means that in the case of a traffic spike, the first five requests would be served as normal but anything beyond that 5th request will be queued up until the first five had been served. This will of course needlessly slowing the site down for visitors.

Issues like this are often flagged up in server logs, letting you know when you hit these configured limits, so it is good to keep your eyes on them. These logs can also indicate that your site is getting busier and help you to grow your infrastructure in good time, along with your visitors.

You might be thinking, “why are there these arbitrary limits getting in my way? I don’t need these at all”.

Well, it is good to have these limits so that in the case of an unusual traffic spike, everything will run slowly but importantly it will work! If they are set too high, or not set at all, you might reach the aforementioned “physical limits” issue risking data corruption and crashing.

Did you know, by default NGINX only runs with one single threaded worker!

Providers

As a small business, it is normally impossible to do everything in house – and why would you want to,  when you need to focus on your business?

So it is good to step back every once in a while and document your suppliers.

Even if you only own a simple website, suppliers could include:

  • Domain registrar (OpenSRS, Domainbox, …)
  • DNS providers (Route 53, DNS Made Easy, …)
  • Server hosting (Rapidswitch, Linode, AWS EC2, …)
  • Server maintenance (Dogsbody Technology, …)
  • Website software updates (WordPress, Magento, …)
  • Website plugin updates (Akismit, W3 Total Cache, …)
  • Content Delivery Network (Cloudflare, Akamai, …)
  • Third parties (Sagepay, Worldpay, …)

All of these providers need to keep their software and/or services up to date. Some will cause more impact on you than others.

Planned maintenance

Looking at server hosting, all servers need maintenance every now and again, perhaps to load in a recent security update or to migrate you away from ageing hardware.

The most important point here is to be aware of it.

All reputable providers will send notifications about upcoming maintenance windows and depending on the update they will let you reschedule the maintenance for a more convenient time – reducing the effect on your business.

It is also good to have someone (like us) on call in case it doesn’t go to plan. Maintenance work might start in the dead of night, but if no one realises it’s still broken at 09:00, heads might roll!

Unplanned maintenance

Not all downtime can is planned. Even the giants, Facebook and Amazon have unplanned outages now and again.

This makes it critical to know where to go if your provider is having issues. Most providers have support systems where you can reach their technical team. Our customers can call us up at a moments notice.

Another good first point of call is a provider’s status page, here you can see any current (as well as past or future) maintenance or issues that are occurring. For example if you use Linode you can see issues on their status page here.

Earlier this year, we developed Status Pile a webapp, which combines provider status information into one place, making it easier for you to see issues at a glance.

Uptime monitoring

This isn’t really a warning sign, but it’s impossible to foresee everything. The above areas are great places to start, but they can’t cover you for the unexpected.

That’s where uptime monitoring comes in. Regardless of the cause, you need to know when your site goes down and you need to know fast.

We recommend monitoring your website at least minutely with a provider like Pingdom or AppBeat.

Proper configuration

Just setting up uptime monitoring is one thing, but it is imperative to configure it properly. You can tell someone to “watch the turkey in the oven” and so they watch the turkey burn!

I’ve seen checks which make sure a site returns a webpage, but if that page says, “error connecting to database” it doesn’t matter!

Good website monitoring checks the page returned includes the correct status code and site content. Perhaps your website connects to your docker application but only for specific actions then you should test specifically as well.

Are you checking your entire website stack?

Cartoon Dog sat at Table woth fire around him. Next frames he is saying 'this is fine'

Who is responsible?

A key part of uptime monitoring – and a number of other items I have mentioned – is that it alerts the right people and that they action those alerts.

If your uptime alerts flag an outage and they are sent to an accounts team it’s unlikely they’ll be able to take action. Equally if an alert comes in late in the evening when no one is around your site might be down until 0900 the next morning.

This is where our maintenance service comes in. We have a support team on call 24/7, ready to jump on any issues.

 

Phew that was a lot, we handle all of this and more. Contact us and see how we can give you peace of mind.

Feature image by Andrew Klimin licensed CC BY 2.0.

stacked logs

Why there’s nothing quite like Logcheck

Anyone who has been to a techie conference in the last few years will know there are lots of log management tools out there, but none have filled the space Logcheck has in our hearts servers.

Logs are our bread and butter. They store details on everything that happens on any server; each request to each asset on this webpage is logged, every login and email sent. In the case of an outage logs are indispensable to see what happened. If you’re under attack it will be logged. Everything is logged so it is essential to pay attention.

Manually checking all server logs is a slow and arduous task and quickly becomes impossible as you scale up. We actively monitor server logs with Logcheck. Logcheck makes this log monitoring possible across hundreds of servers by reducing the logs needing to be looked at, it does this with the exception tracking approach.

Most log management tools use a blacklist approach, looking for bad words such as “attack”, “bad” and “error”. In doing so they only tell you about the “known bad”, the log lines that have shown errors before. Big problems will come if you’re hosting a brand new app or using new software, there is no way of knowing what is bad and what should be alerted on. You rely completely on the new software to have configured logging that matches your current rules.

Logcheck’s whitelist approach fixes the problem these other tools have, as it passes you all unknown/rogue logs. This makes it perfect for any venture in to the unknown by telling you exceptions to known good rules.

Regex can be used in the whitelist making the rules very customisable and still broad enough to not have to whitelist every single combination of log.  We maintain our whitelist rules on a per-server basis, as logs that are OK on one server could indicate a problem on another.

Logcheck and log administration are services offered in our maintenance packages.

Contact us today to find out more!

Feature image background by gregloby licensed CC BY 2.0.

How to set-up unattended-upgrades

Making sure software is kept up to date is very important.  Especially when it comes to security updates.  Unattended-upgrades is a package for Ubuntu and Debian based systems that can be configured to update the system automatically.  We’ve already discussed manual patching vs auto patching, most of this post will assume you’d like to set-up automatic updates.  If you want complete control of updates you may need to disable unattended-upgrades, see the manual updates section below.

Automatic Updates

Make sure you have installed unattended-upgrades and update-notifier-common (in order to better determine when reboots are required).  On some recent operating systems unattended-upgrades will already be installed.

sudo apt-get install unattended-upgrades update-notifier-common

Once unattended-upgrades is installed you can find the configs in /etc/apt/apt.conf.d/.  The 50unattended-upgrades config file has the default settings and some useful comments.  20auto-upgrades defines that updates should be taken daily. The default configuration will install updates from the security repository

We would suggest creating a new file and overwriting the variables you want to set rather than changing files that are managed by the package.

You can create the following as /etc/apt/apt.conf.d/99auto-upgrades:

# Install updates from any repo (not just the security repos)
Unattended-Upgrade::Allowed-Origins {
"*:*";
};
# Send email to root but only if there are errors (this requires you to have root email set-up to go somewhere)
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::MailOnlyOnError "true";
# Remove packages that should no longer be required (this helps avoid filling up /boot with old kernels)
Unattended-Upgrade::Remove-Unused-Dependencies "true";
# How often to carry out various tasks, 7 is weekly, 1 is daily, 0 is never
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
# Use new configs where available but if there is a conflict always keep the current configs.
# This has potential to break things but without it the server won't be able to automatically update
# packages if you have changed a configuration file that the package now has an updated version of.
Dpkg::Options {
"--force-confdef";
"--force-confold";
}
# Some updates require a reboot to load in their changes, if you don't want to monitor this yourself then enable automatic reboots
Unattended-Upgrade::Automatic-Reboot "true";
# If you want the server to wait until a specific time to reboot you can set a time
#Unattended-Upgrade::Automatic-Reboot-Time "02:00";

Have a look at the comments but the key things to point out here are:

  • The above config will install all updates.  You can define what repositories to update from as well as any packages to hold back but then you will obviously end up with some software out of data.
  • It is important to make sure you will be informed when something goes wrong.  One way to do this is to send errors to root and have all root email sent to you (you can define this in /etc/aliases).  To test you are receiving email for root you can run:
    echo "Test email body" | mail -s "Test email subject" root
  • If you aren’t going to follow security updates and decide when to reboot your server make sure you have automatic reboots enabled, it is probably worth setting an appropriate reboot time.

Manual updates

If you want to manually update your server then there is no need for you to install unattended-upgrades.  However as some operating systems have it pre-installed so you may have to disable it.  The easiest way to disable unattended-upgrades is to create the following as /etc/apt/apt.conf.d/99disable-auto-upgrades:

APT::Periodic::Unattended-Upgrade "0";

Feature image by Les Chatfield licensed CC BY 2.0.

Holey jeans

Manual patching vs auto patching

Everyone agrees keeping your software and devices updated is important.  These can be manually or automatically installed.  People assume that automatic is the better option however both have their advantages.

I’m Rob, I look after maintenance packages here at Dogsbody Technology. I want to explain the advantages between the two main patching approaches.

What to Patch

Before we get into the differences of how to patch it’s worth discussing what to patch.

Generally speaking we want to patch everything.  A patch has been produced for a reason to either fix a bug or security issue.

Sometimes patches add new features to a package and this can be when issues occur.  Adding new features can cause things to break (usually due to broken configuration files).

Identifying when a patch release is for a bug, a security fix or adding a feature can be hard. In some cases the patch can be all three things.  Some operating systems try and separate or tag security patches separately however our experience shows that these are rarely accurate.

One of the reasons we like manual patching so much is that it allows us to treat each patch/customer/server combination independently and only install what is required, when it is required.

Auto Patching Advantages

The server checks and updates itself regularly (hourly/daily/weekly).

  • Patches can easily be installed out of hours overnight.
  • Patches are installed during the weekend and bank holidays.
  • Perfect for dev environments where downtime is OK.
  • Perfect for use in Constant Integration (CI) workflows where new patches can be tested before being put into production.

Our automatic patching strategy is to typically install all patches available for the system as it is the only sure way to know you have all the security patches required.

Manual Patching Advantages

A notification (e-mail or internal ticket) is sent to the server admin who logs onto the server and installs the latest updates.

  • Patches can be held during busy/quiet periods.
  • The admin can ensure that services are always restarted to use the patch.
  • The admin can search for dependant applications that maybe using a library that has been patched (e.g. glibc patches)
  • The admin is already logged onto the server ready to act in case something does break.
  • Kernel reboots (e.g. Meltdown or Stack Clash) can be scheduled in and mitigated.
  • Configuration changes can be reviewed and new options implemented when they are released. Catching issues before something tries to load a broken configuration file.
  • Perfect for production environments where you need control. Manual patching works around your business.

Because we manually track the packages used by a customer we can quickly identify when a patch is a security update for that specific server.  We typically patch security updates on the day it is released also install non-security updates at the same time to ensure the system has the latest and greatest.

 

Are you unsure of your current patch strategy? Unsure what the best solution is for you? Contact us today!

 

Feature image background by Courtnei Moon licensed CC BY 2.0.

DROWN vulnerability

Dogsbody Technology maintenance customers are already protected against the newly disclosed DROWN attack, but as of the 1st March, 33% of all HTTPS sites are affected by this vulnerability.

The DROWN (Decrypting RSA with Obsolete and Weakened eNcryption) vulnerability affects HTTPS and other services that rely on SSL and TLS, these cryptographic protocols that make security over the Internet possible.

The attack affects all SSLv2 servers and allows attackers to decrypt HTTPS traffic during transfer letting them spy on traffic. In some cases encryption can be broken within minutes!

The fix web servers is to disable SSLv2 support:

  • For Apache: SSLProtocol all -SSLv2 -SSLv3
  • For Nginx: ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

For more information on the attack and research paper take a look at the official DROWN Attack website.

Dogsbody Technology are Linux SysAdmin’s, building secure scalable reliable servers for the internet. We keep our servers up-to date and in doing so have already mitigated this attack.

If you want your site checked or have any questions please contact us.