Using Brotli compression to reduce CDN costs

This article appeared originally on Mozilla Marketing Engineering & Operations blog

The Snippets Service allows Mozilla to communicate with Firefox users directly by placing a snippet of text and an image on their new tab page. Snippets share exciting news from the Mozilla World, useful tips and tricks based on user activity and sometimes jokes.

To achieve personalized, activity based messaging in a privacy respecting and efficient manner, the service creates a Bundle of Snippets per locale. Bundles are HTML documents that contain all Snippets targeted to a group of users, including their Style-Sheets, images, metadata and the JS decision engine …

in software · Tue 21 August 2018

NeReS: Manage NewRelic Synthetics Monitors from the command line.

NewRelic Synthetics is a service to monitor websites that comes free of charge with all accounts and offers a satisfying set of features:

• Website up-time monitoring from multiple locations around the world
• Content validation
• SSL certificate validation
• Email alerting on error

I developed NeReS, an (unofficial) cli tool to manage monitors from the command line.

NeReS can list the current account monitors, add new, edit or delete existing ones. It can also return the information in raw JSON format so you can further process and automate your monitor generation using tools like jq.

Using NeReS you can automate the generation …

in software · Fri 03 November 2017

Automating Podcast generation from SoundCloud

There's this popular daily FM Radio show in Greece which posts their shows on SoundCloud after broadcasting them. It's a good -albeit not great, just HTML5 audio is fine- way to listen the show on demand if you're on Desktop. The website is not mobile friendly and the whole embedded SoundCloud experience is sub-optimal. Let alone that you cannot just add the feed to your favorite podcast player to enjoy it.

There's an RSS feed on iTunes but it's manually updated and inevitably lags a day or two behind, depending on the availability of the maintainer.

I decided to fix …

in software · Tue 10 October 2017

Monitoring python cron jobs with Babis

Over at MozMeao we are using APScheduler to schedule the execution of periodic tasks, like Django management tasks to clear sessions or to fetch new job listings for Mozilla Careers website.

A couple of services provide monitoring of cron job execution including HealthChecks.io and DeadManSnitch. The idea is that you ping a URL after the successful run of the cron job. If the service does not receive a ping within a predefined time window then it triggers notifications to let you know.

With shell scripts this is as simple as running curl after your command:

$ ./manage.py clearsessions && curl …

in software · Wed 24 May 2017

January in Athens Python Users Meetup

On 24th of January we had the first Python meetup for 2017 as always in Hackerspace.gr. Based on our typical setup we started with two talks:

• Myself, I kicked off the meetup with a Micropython on ESP8266 talk. I started with quick intros on Micropython and the ESP8266, moved to using the RERL over Serial with minicom and ended spectacularly with blinking neopixels.

• Spyros followed talking about Nameco based microservices and his Tweetmark project to save tweets for later reading and do it in style. Moved on with a live demo and successfully fixed the obligatory demo bug defeating …

in software · Fri 03 February 2017

Getting advantage of docker image cache on 1.13

Last week's release of Docker (v1.13) added the cache-from build flag which allows the use of local images as build cache and solves the layer caching issue created by Content Addressability as I've blogged before.

To use the new feature and take advantage of the caching for faster image builds first pull the image that you want to base you build upon and then build using --cache-from

For example:

docker pull myimage:v1.0
docker build --cache-from myimage:v1.0 -t myimage:v1.1 .

You should always keep in mind though that you must fully trust the registry you're …

in software · Mon 23 January 2017

Taco Bell Parallel Programming

While working on migrating support.mozilla.org away from Kitsune (which is a great community support platform that needs love, remember that internet) I needed to convert about 4M database rows of a custom, Markdown inspired, format to HTML.

The challenge of the task is that it needs to happen as fast as possible so we can dump the database, convert the data and load the database onto the new platform with the minimum possible time between the first and the last step.

I started a fresh MySQL container and started hacking:

Load the database dump

Kitsune's database weights about …

in software · Thu 01 December 2016

Systemd Unit to activate loopback devices before LVM

In a Debian server I'm using LVM to create a single logical volume from multiple different volumes. One of the volumes is a loop-back device which refers to a file in another filesystem.

The loop-back device needs to be activated before the LVM service starts or the later will fail due to missing volumes. To do so a special systemd unit needs to be created which will not have the default dependencies of units and will get executed before lvm2-activation-early service.

Systemd will set a number of dependencies for all units by default to bring the system into a usable …

in software · Mon 17 October 2016

Docker cache on Travis and Docker 1.12

Edits

2016-09-12 Edited to prevent caching of builds in pull requests.
2017-07-07 Edited to point to exact git commit b/c we don't use Travis for Snippets Service anymore.

I blogged before about building Docker images on Travis and suggested uploading images after successful test runs to Docker Hub and use them as Cache after downloading them in next Travis runs.

Travis upgraded recently to Docker version 1.12 (from 1.9) and since version 1.10 Docker features Content Addressability for layers. This change breaks caching and we need to implement a workaround using Travis cache.

Changes need to …

in software · Thu 01 September 2016

User friendly website analytics with Sandstorm Oasis and Piwik

Piwik is a great FLOSS website analytics platform. I've been self hosting it for different small websites I've managed through the years. Although it's fairly easy to setup and maintain at this level of use, I want to avoid having another service in my maintenance list.

While looking for user and web respecting alternatives -read looking for something else than Google Analytics- I realized that Sandstorm Oasis does support Piwik.

I logged-in and setup my Piwik instance, or Grain as Sandstorm calls instances, in less than 30 of seconds. The tricky part is to copy the code provided by Sandstorm …

in software · Tue 26 July 2016

Build and Test against Docker Images in Travis

The road towards the absolute CI/CD pipeline goes through building Docker images and deploying them to production. The code included in the images gets unit tested, both locally during development and after merging in master branch using Travis.

But Travis builds its own environment to run the tests on which could be different from the environment of the docker image. For example Travis may be running tests in a Debian based VM with libjpeg version X and our to-be-deployed docker image runs code on-top of Alpine with libjpeg version Y.

To ensure that the image to be deployed to …

in software · Fri 17 June 2016

Remove users from git-crypt enabled repository

Git crypt is a neat git extension to encrypt some files - if not all - in a git repository. Integrates nicely with git using filters and it's use is completely transparent once you have unlocked a repository.

Using git-crypt you can still share a repository in public and maintain a set of files with secrets that are accessible to a limited number of users. Especially useful for open source projects.

At some point maybe you'll need to remove one of the users who have access to the encrypted files. Git-crypt does not provide a command to remove users (yet) because it's …

in software · Tue 24 May 2016

Takis - A util that blocks until a port is open.

Over at Mozilla's Engagement Engineering we use Docker to ship our websites. We build the docker images in CI and then we run tests against them. Our tests usually need a database or a cache server which you can get it running simply with a single command:

docker run -d mariadb

The problem is that this container will take some time to initialize and become available to accept connections. Depending on what your test and how you run your tests this delay can cause a test failure to due database connection timeouts.

We used to wait on executing our tests …

in software · Mon 16 May 2016

Static site hosting on Dokku and Deis

Deis and Dokku, the open source Heroku-like PaaS, can be used for hosting static sites too. Since they both support Dockerfile based deployments all we need is an Docker Image with Nginx.

I created giorgos/dokku-static-site which uses the ONBUILD instruction. To use it create a Dockerfile at the root directory of your static site with only one line:

FROM giorgos/dokku-static-site

and then place all your files under the html directory. If moving your website files to another directory isn't in your plans, you can alternatively create a symbolic link

ln -s . html

Then push to your Deis / Dokku …

in software · Mon 09 May 2016

Host your own music server with supysonic

I like to self-host services to make things fit my needs instead of the other way around and to satisfy my need to control my data. Also remember that self hosting is really important for the health of the open web and internet.

There are a couple of options for self-hosted media servers. I always go for FLOSS solutions, so Plex and friends are not an option. A couple of years ago I tried Subsonic, including the MadSonic and Musicabinet forks. Subsonic looked really outdated at that time, so that didn't last long. Musicabinet was second in my evaluation and …

in software · Wed 27 April 2016

Programming Pro Trinket with an Arduino

Pro Trinket is a ATmega328 based board by Adafruit. It's compatible with Arduino IDE and most of the code written for Arduino. It's small factor, cheap and comes with neat accessories like this LiPo backpack addon.

Pro Trinket is cheaper, smaller and consumes less power than a typical Arduino board because it lacks things like the FTDI chip used to communicate between the programmer and the board.

To program the Pro Trinket you need to have an external FTDI board or figure out how to make programming over USB work with Linux. I opted for the first option and abused …

in hardware, · Sat 31 October 2015

Downloading HDS Flash (f4m) video streams.

Some WebTV or other video streaming websites use a special Adobe (surprise, surprise) inspired protocol to stream their content. The special thing about this protocol is that the file is split into chunks which I guess is good if you want to jump on different points in the video or you have an unstable internet connection but really annoying if you just want to download the video and view it in a different player.

This protocol is called Abode Dynamic Streaming for Flash and usually the sites link to f4m manifest files. From the SWF Wikipedia Page we read get …

in software · Wed 21 October 2015

Fleet job to remove unused docker images

Engagement Engineering, the team that I'm part of at Mozilla, runs two Deis clusters on AWS to host important websites including www.mozilla.org.

Deis is a Heroku-inspired PaaS which utilizes CoreOS and Docker. It's a great open-source project, developed in the public, with a great Community and commercially backed by Engine Yard.

Apps on Deis run within Docker containers which run on CoreOS machines that form the Deis cluster. Each new release of your code, i.e. each new deis pull or git push, creates a new Docker image that is stored in the internal Deis Docker Registry and …

in software · Tue 13 October 2015

List the extensions of files under a directory

I want to list the extensions of filenames in a directory, sorted and unique.

First list all the files, leaving out directories using:

find -type f .

Then use awk to get the last three characters of each line, i.e. filename.

find . -type f | awk '{ print substr( $0, length($0) - 3, 4) }'

AWK's substr function extracts four characters (last argument) of string $0 (first argument) starting at full lenth of the string minus three characters (second argument).

This will list the extensions from all files in the current directory and all directories below.

Now sort and uniq the output of …

in software · Mon 15 June 2015

Duplicity scp backend weirdness

I'm a big fan of Duplicity and I use to backup my laptop to my office based server over scp which in its turn encrypts everything with EncFS and syncs everything up to my SpiderOak account. I was going under a typical maintenance today and I run duplicity collection-status to see how many backup sets I have.

Surprisingly duplicity returned only one full backup set more than a year old and no other backup sets. After some digging I found that the returned backup set was indeed the only full set I had but besides that I had many incremental …

in software · Tue 28 April 2015