Categories
Uncategorized

Practising code with Exercism

During the Coronavirus lockdown, I’ve been doing a programming exercise on Exercism every few days.

Level up your programming skills with 3,355 exercises across 50 languages, and insightful discussion with our dedicated team of welcoming mentors. Exercism is 100% free forever.

What is Exercism?

I started with their PHP track a few years ago and completed exercises on-and-off, but lately I’m enjoying the Python track. Here are a few of my most recent solutions:

Once you’ve finished an exercise, you can view other programmers’ solutions, or get feedback from mentors who are further along with your chosen language.

The Exercism people are working on some good improvements for version 3, one of which is improving the UI around mentor-student communication. As a result of mentors’ involvement in that work, mentorship is slower than normal, but it remains one of my favourite things about the platform.

Categories
Uncategorized

Trying Cloudflare

I installed the free version of Cloudflare on this blog in order to understand the setup involved. The advertised benefits of Cloudflare are:

  1. Security, e.g. mitigating denial-of-service attacks, identifying bots and limiting scrapers. These measures reduce workload on your own infrastructure and protect your content.
  2. Performance, including an easy-to-use (and often automatic) content delivery network (CDN) to serve files quickly from diverse geographical locations closer to each user;
  3. Reliability, by increasing redundancy in responding to DNS queries and routing traffic efficiently on distributed infrastructure.

I don’t have any problems with bots or scalability on this blog (I can dream) but a little extra performance never rarely hurts. Mostly it’s a learning experience ahead of getting Cloudflare up and running on more important websites.

If my past CDN experience is anything to go by, re-routing all traffic always comes with a few challenges but, for a simple WordPress site like this one, the setup was simple. The main step was to point the domain nameservers to Cloudflare instead of my original DNS provider. It felt incredibly simple.

The remaining quirks were around user IP addresses and cache invalidation.

Ensuring that WordPress sees the end user’s IP address

Since HTTP(S) requests from users’ browsers go to Cloudflare, and Cloudflare makes corresponding (less frequent) requests to my server, WordPress would see Cloudflare’s IP addresses every time instead of the IP addresses of individual end users.

This matters, for example, because WordPress saves the IP address of a user when they add a comment, and Akismet relies upon it to help identify spam.

Cloudflare helpfully passes the original user’s IP address in an HTTP request header, but I needed a way to tell WordPress to look for that header. There are two Apache modules which help:

  1. mod_cloudflare is mentioned in some old Cloudflare documentation, but it’s no longer supported;
  2. mod_remoteip is recommended now although it isn’t Cloudflare-specific and requires some configuration.

However, instead of solving the problem at that low level, I realised there was a Cloudflare plugin for WordPress itself. Not only did it solve the IP address issue, but it also met my another requirement…

Clearing the CDN cache when something changes

If all your static files (images, CSS, JavaScript) are cached around the world on Cloudflare’s CDN, you need a way to tell them to clear those caches when you change something.

You can do this manually in Cloudflare’s website, which would probably be okay for a site this small. Alternatively, you can send an API request with the same instruction. Best of all, the process can be automated fully by the Cloudflare WordPress plugin above, which gave it the edge over the Apache modules.

Categories
Uncategorized

Backing up WordPress to S3

This blog is hosted on a virtual machine at Linode, who provide a backup facility of their own. I’m using that, but I think it’s worthwhile to push an additional snapshot somewhere else at least once a week for additional redundancy.

I already have a personal account at Amazon Web Services, so each week, I’m sending a database dump and a tarball of the WordPress filesystem to an S3 bucket.

The high level steps:

  1. Create an S3 bucket taking care not to allow public access.
  2. Configure the bucket’s Lifecycle Policy to expire old backups after 95 days.
  3. Create an Amazon IAM user and grant API access to the S3 bucket, but not to any other resources on the AWS account.
  4. Install s3cmd on the Linode box where WordPress is hosted. It’s a command line interface for S3, written in Python.
  5. Deploy a shell script to create the filesystem tarball and dump the MySQL database before pushing both files to S3. Guy Rutenberg published a simple, but effective, WordPress backup script in 2008. I added a new command at the end to clean up the local copies of the backup files once they’re pushed successfully to S3.
  6. Download phusion-server-tools to the Linode box. These include a script called silence-unless-failed. Used as a wrapper around cron jobs, it suppresses output unless a script exits with an error code. Combined with cron’s own MAILTO command, it generates email alerts when something goes wrong, but only then.
  7. Create the cron job. I think weekly is enough to balance risk, storage cost and my blogging frequency. I’m relying on Linode’s backups in the first instance, so the S3 backups will only become important if something really bad happens.

Here’s Guy’s shell script, with the new cleanup command at the end:

#!/bin/bash

# (C) 2008 Guy Rutenberg
# This is a script that creates backups of my blog.

DB_NAME=database_name
DB_USER=database_user
DB_PASS=****
DB_HOST=db_host

#no trailing slash
BLOG_DIR=/path/to/blog
BACKUP_DIR=/path/to/backups
S3BUCKET=s3://bucket-name/

# end of configuration - you probably don't need to touch anything below
DUMP_NAME=${DB_NAME}-$(date +%Y%m%d).sql.bz2

echo -n "dumping database… "
mysqldump --user=${DB_USER} --password=${DB_PASS} --host=${DB_HOST} ${DB_NAME} \
| bzip2 -c > ${BACKUP_DIR}/${DUMP_NAME}
if [ "$?" -ne "0" ]; then
    echo "failed!"
    exit 1
fi
echo "done"

TAR_NAME=${BLOG_DIR##/}-$(date +%Y%m%d).tar.bz2
echo -n "Creating tarball… " tar -cjf ${BACKUP_DIR}/${BLOG_DIR##/}-$(date +%Y%m%d).tar.bz2 ${BLOG_DIR}
if [ "$?" -ne "0" ]; then
    echo "failed!"
    exit 1
fi
echo "done"

echo -n "Uploading SQL dump to Amazon S3… "
s3cmd put ${BACKUP_DIR}/${DUMP_NAME} ${S3BUCKET}${DUMP_NAME}
if [ "$?" -ne "0" ]; then
    echo "failed!"
    exit 1
fi
echo "done"

echo -n "Uploading tarball to Amazon S3… "
s3cmd put ${BACKUP_DIR}/${TAR_NAME} ${S3BUCKET}${TAR_NAME}
if [ "$?" -ne "0" ]; then
    echo "failed!"
    exit 1
fi
echo "done"

# additional command to delete local copies
echo -n "Deleting local copies of SQL dump and tarball… "
rm ${BACKUP_DIR}/${TAR_NAME} ${BACKUP_DIR}/${DUMP_NAME}
if [ "$?" -ne "0" ]; then
    echo "failed!"
    exit 1
fi
echo "done"
Categories
Uncategorized

We live inside now

Everyone is inside because of COVID-19. We live in interesting times. So far, I’m still programming from home as normal, despite a lot of big changes for everyone else in the business where I work.

TOTP is a time-based one-time password algorithm

Our Magento sites have two-factor authentication on the admin login screens, and it is only required outside the office network. If you’re at home, you need your username, your password and a code which changes a few times every minute.

Everyone in the business who’s now working from home, but who didn’t in the past, needs to be running an app on their phone to get those codes. When you set that up a few times, especially for other people, you hit every known problem with TOTP.

For example, you spend a long time working out why the code isn’t working, only to realise that the user is trying the wrong password. It’s common for users not to know their passwords (and not to know that they don’t know them) when they’re accustomed to using browser auto-fills.

Or: another user’s mobile phone clock was off by 36 seconds, so the generated codes were wrong. Visiting time.is from the device helps; it shows how your own clock compares to that of an atomic clock, somewhere, and it (the website, not the atomic clock!) will apparently “normally have a precision of 0.02-0.10 seconds”. Fix the time on their phone, and all is well.

SOAP, Python and Zeep

Everyone’s talking about soap at the moment, but I’m talking about SOAP.

I migrated one of our shipping carrier integrations from one SOAP API to another. The Zeep Python library and the service’s WSDL URL meant that I didn’t have to spend too much time looking at raw XML.

But let’s be honest: I still don’t fully understand how SOAP works, even though this particular integration is now up and running smoothly.

Debugging outbound mail from your web application

If you’re a developer, this is one of those things you’ve either been doing forever, or one of those things you will want to do forever.

When you’re running a dev environment, you need to see which emails come out of it, without actually sending them to anyone. MailCatcher can help.

First, you tell your application to connect to MailCatcher instead of connecting to an SMTP server whenever it’s time to send an email. The outgoing emails are intercepted and presented in a simple web interface, ready to read and debug.

I can’t speak to its usefulness in testing the layout or design of the emails, but it certainly makes it easier to debug their content or to verify that they’re definitely being sent in the first place.

What’s even more important: if you do have any real email addresses in your development environment, it helps you avoid sending emails to real people by mistake.

I run Docker dev environments, so I added a new container to each of them based on an existing MailCatcher image I found on Docker Hub.

I have large SQL scripts for each Magento website which I use to sanitise the production data before loading it into my dev database. I adjusted those scripts, adding a few simple queries to point Magento’s SMTP settings to the MailCatcher containers.

Now, when I pull data from production to development, it gets sanitised and the outgoing emails get routed to MailCatcher without me having to think about it.

It’s nothing too new or shiny, but it’s one of those changes that sticks around in dev envs for years; one less fiddly distraction to worry about in numerous ongoing projects.

Who doesn’t dislike Magento 1.x upgrades?

Magento 1 is almost end-of-life but many merchants are still running it and making various plans to keep running it securely after the deadline. The move to Magento 2, or another platform entirely, is a big project, especially if you’ve built a lot of bespoke modules, integrations and theme changes on top. Even if you’ve done it the right way, without modifying the core.

In the meantime, we still see an occasional version bump within Magento 1.

Ahead of a move to PHP 7.2 (long overdue), I’m upgrading from 1.9.0.1 to 1.9.4.3. It’s a time-consuming process, but I was pleased to see that it’s not just me who feels that way.

Categories
Uncategorized

Unread books

I gathered and counted sixty-seven of them, split almost evenly between Apple Books, Kindle and print. They all still sound as interesting as when I bought them. I won’t need any more books for a while but I don’t need any more self-imposed rules either.

Appropriately, I’m starting with Nicholas Carr’s The Shallows: How the internet is changing the way we think, read and remember.

I crammed for exams in the library’s cavernous reading room, looked up facts in the weighty volumes on the reference shelves, and worked part-time checking books in and out at the circulation desk. Most of my library time, though, went to wandering the long, narrow corridors of the stacks. Despite being surrounded by tens of thousands of books, I don’t remember feeling the anxiety that’s symptomatic of what we today call “information overload.” There was something calming in the reticence of all those books, their willingness to wait years, decades even, for the right reader to come along and pull them from their appointed slots. Take your time, the books whispered to me in their dusty voices. We’re not going anywhere.

Nicholas Carr, The Shallows (2010), chapter one

Nicholas goes on to describe how our tools become a part of us; our brains come to treat them as extensions of our own physical and mental capabilities.

Unlike a lot of recent writing about (and on) the internet, The Shallows so far successfully avoids the CGP Grey trap:

Most arguments about medium compare the best of the old with the worst of the new.

CGP Grey [citation needed]

I’ve accepted the inevitable: some of the sixty seven will forever remain entirely—or just partially—unread. At least by me, at least for now.

Categories
Uncategorized

Thinking about progressive web apps

The Magento ecosystem has been talking about progressive web apps for ages now with various levels of (enthusia|cynici)sm so, when I saw a tutorial on freeCodeCamp about the basics, I gave it a quick try.

As someone whose last major frontend project used JavaScript ES5 on AngularJS (before it became just Angular), I had to park the urge to dig into arrow functions and CSS Flexbox when they cropped up in the sample code.

I have two ideas I think might be a good fit for PWAs. One is a personal learning project where I track my fuel spend every time I fill the car and sync that up to a server. That’d let me get up to speed on modern CSS/JS as well as the PWA stuff. Another is for my day job, and involves iOS, large photo files and locations with poor-to-no data signal.

I quickly learned that Android embraces PWAs much more enthusiastically than Apple do, even though the earliest ever iPhone apps were built on web technologies. Since the relevant work devices are all iOS, this matters. With App Store’s revenue-generating walled garden, it’s easy to see why Cupertino isn’t rushing to improve their support, although there are improvements in the latest few releases. The limitations on capabilities remain significant: 50 MB of cache storage; caches being cleared unexpectedly to free up resources; and no background sync.

I think the fuel-tracker fun is a go, but the work project will need something native.

Categories
Uncategorized

That Safari thing

Last week, I thought I’d found a good Safari tip to open links from emails in the current macOS Space instead of having them switch me to another Space which already has a Safari window.

It didn’t work well, and I’ve switched it back. When a website decides a link should open in a new window, with this setting enabled, it really did open in a new window instead of a new tab. Since there’s no way, just by looking at a link, to know that the website will do this, I’d have to right click on every link and choose explicitly what I want. That’s no use.

Categories
Uncategorized

Terrible code

I found some terrible code: no regard for data types, instead relying on the magic of implicit casting and type conversion; variables not named to indicate the units of the values within. It’s a pity I wrote it.

“If I had a dime for every time I’ve seen someone use FLOAT to store currency, I’d have $999.997634”

Bill Karwin

Enough said.

macOS Spaces and Safari

I sometimes have multiple projects open in different virtual desktops—macOS “Spaces”—at the same time.

In Mission Control preferences, there’s a setting: “When switching to an application, switch to a Space with open windows for the application.” I’ve switched that off, so if I Cmd-Tab to Safari from within a project Space and there’s no Safari window in that Space, I can use Cmd-N to open one.

It works well, but it’s not seamless. For example, my first Space is for general stuff (email, Slack, todo list, etc.). When I click a link in an email, macOS takes me to the most recent Safari window, in another space, and opens the link as a new tab. It’s rarely the one I want.

I found the an answer in the Safari Tabs preferences:

Open pages in tabs instead of windows: Never

Now, when I click a link in another application, it opens in a new window in the current space. If I want it as a tab in a specific project desktop, I right click and copy the link’s URL instead, and open the tab manually where I want it.

It’s the little things.

23andMe and Ancestry

In 2015, I sent a small vial of saliva to 23andMe. After numerous warnings and “Are you really sure?” messages, they revealed what they say are my genetic mutations that might put me at risk for various diseases. Nothing interesting came up.

They told me that my ancestry was very British/Irish and very Northern European. No surprises there either, so I didn’t spend much time on the site after that.

Then, a few weeks ago, I received a message from a DNA relative in the USA. Exciting! We share a set of great-great grandparents who lived a few miles from where I grew up. It took some days to figure out who the common connection was and, along the way, we learned about some great aunts and uncles we didn’t know existed.

I’m curious now about the several hundred other DNA relatives listed on 23andMe, and I wish they’d put some more detail—even a list of towns or family surnames—on their profiles.

I signed up for a trial on Ancestry.co.uk too, which is fascinating. After plugging in all the names, birth dates, death dates and locations, it searches lots of public archives for people you’ve entered and helps you find new connections very quickly via those records. It’s amazing when your own tree intersects with those uploaded by other users, some of whom have also uploaded old photos.

I know very little about genealogical research, but I’m learning slowly. For example, I’ve realised that census records (1901 and 1911 in Ireland especially) are helpful because they place parents and children in one document, with birth dates. Other records aren’t so useful with many people’s birth dates listed only approximately. It turns out that not everyone knows, or knew, their own dates of birth.

This post from the Lavazza cafe at the M1 services outside Lisburn, Northern Ireland after dropping my parents to the airport. It’s quiet.

“Looking for somewhere a bit quieter?”
Categories
Uncategorized

Spam > Content

Akismet is good

Just five. That’s the number of spam comments required per day on this almost-empty blog to make me want to have Akismet running. If you’re not familiar, it’s a third party service that separates real comments from spam similar to how your email provider might remove (most) spam from your inbox. It comes installed on WordPress by default, but needs to be activated with an API key before it gets to work.

Since I last installed a fresh copy of WordPress, I see they’ve introduced a default privacy policy page; probably around the time of all the GDPR action. As I’m collecting very limited data it was quick to edit the text for relevance before publishing.

I was tempted to stick Google Analytics on this site too, but I don’t need to know that nobody is reading yet. I’m determined to build a writing habit for now—whatever it produces. Lucky you!

A week in Windows

I’ve been all-in on Apple stuff since 2010, a full six years before Microsoft released Windows Subsystem for Linux. Back then, I wanted a Unix-style command line environment for developing PHP applications. Now, in 2020, with all my Apple computers and devices loaded up with good software and talking to each other in all the right ways, I’m hooked.

This week, I found myself migrating an old .Net application from one server to another. Very glad that I’d taken detailed notes the first time I deployed it, I couldn’t help also being glad that Windows Server is a graphical interface and much more intuitive to learn than the command line.

I do have a Windows 10 virtual machine running on Parallels Desktop. In a Microsoft mood, I played with that for a few hours too, getting up to speed on all the configuration options and generally how to get around. I hadn’t realised that Microsoft have rebuilt their Edge browser on top of Chromium, and that it’s being rolled out gradually to replace the legacy version.

Google Nest Minis

YouTube sent us two free speakers at home for being YouTube Premium customers. They’re up and running and much better than our first-gen Amazon Echo at hearing us over the TV and everyday household noise. So far, all our requests have been about music, food timers and fact-checking.

People are very concerned about privacy with these speakers. I’ve never worried too much about it. I’m happy to exchange some data with huge and competent companies, in an educated way, in return for useful functionality and more relevant ads.

Their data security is much better than mine will ever be. I take them at their word that they’re mostly doing only what they say they’re doing—usually—and, if I’m going to get ads at all, I’d like them to be for things I might want to buy.

I hope I’m right.

By the way: On the subject of data privacy, if you’re a heavy Google user and have never visited the My Google Activity dashboard, you’re in for a treat.

Categories
Uncategorized

Speeding up Magento Docker dev env

I have several Magento 1.9 development environments running on Docker Compose and Docker Desktop for Mac, both of which have always been slow. Every page reload took 6-10 seconds which really hit productivity.

A friend pointed me to Docker’s settings for bind-mount consistency. By default, when a file is written inside the container, the system waits for that change to be persisted to the host before continuing. The opposite happens for changes written to the host. It’s very quick, but if there are hundreds of changes, it noticeably affects performance.

If you’re willing/able to settle for eventual consistency, things move more quickly. I switched the MySQL data volume to “delegated”, which means the version inside the container is authoritative. That’s where all the MySQL changes happen.

I also switched the Magento filesystem, this time to “cached”. With “cached”, the files on the host system (macOS) are authoritative. It seems unlikely that I’ll manage a manual browser refresh before the macOS change is persisted into the container!

There’s more info about performance of shared volumes in the Docker docs.