Data delay on dashboards.
Sep 19, 00:00 PDTResolved - The backlog has cleared, your stats are fast again.IdentifiedThere is currently a delay in data appearing on your dashboard. Your data hasn't been lost, it's just delayed....
View ArticleDashboard issues
Oct 6, 00:00 PDTResolved - ResolvedEverything is back to normal. We were encountering rate limiting from a service that the dashboard relied on (AWS Parameter Store). We have increased the throughput...
View ArticleLayer 7 DDoS attacks
Nov 8, 00:00 PSTResolved - The attack ended November 7th PT. As we stated before, only 0.01% of customers were hit with spam, but all customers were affected.The attack was intermittent and typically...
View ArticleLayer 7 DDoS attack
Nov 14, 00:00 PSTResolved - The attack on 14th November lasted around 3 hours, and our data collector was up & down, but we didn't want to update this page until we'd finished implementing our new...
View ArticleLayer 7 DDoS attack
Nov 21, 00:00 PSTResolved - We've been under DDoS attacks for a while now, on and off. Today we had our first win. We engaged our 24x7 DDoS team, they identified patterns in the attack and were able to...
View ArticleProvider issues
Nov 25, 10:07 PSTResolved - What a day. AWS having issues like this, where it actually makes mainstream news, is unbelievably rare. Having said that, it's given us some things to think about. We can't...
View ArticleSpam/DDoS attack
Apr 3, 00:00 PDTResolved - Today, we tried out some new rules that were meant to reduce the strictness of our firewall. Unfortunately, the rule change meant huge amounts of spam was able to pass...
View ArticleInfrastructure upgrades
Oct 11, 00:00 PDTResolved - We performed some upgrades to our default Fathom script (cdn.usefathom.com) on the 11th October 2021. We moved the ingest endpoint to a new CDN, with added security and...
View ArticleLetsEncrypt Bug
Feb 1, 00:00 PSTResolved - Two customers using our old custom domain system (deprecated last year) ran into issues with SSL certificates being revoked by LetsEncrypt, our old certificate provider.We...
View ArticleAWS Downtime
Mar 9, 00:00 PSTResolved - Parts of us-east-1 appeared to have fallen offline for around 7 minutes. During this time, our services were only partially available (we use multiple clouds to process...
View ArticleDDoS Attack
Mar 7, 10:04 PSTResolved - Today’s attack was unique because it was completely unintentional.There was a problem with a customer’s site, because they had programmed an infinite loop on their event...
View ArticleCustom domain migration
Mar 24, 09:45 PDTResolved - Last night we performed a migration on the way custom domains work. Historically, any IP blocking you did or allowing of domains was implemented at a network level (on the...
View ArticleDDoS Attack
Jun 7, 10:00 PDTResolved - Today’s attack was unique because it was completely unintentional.There was a problem with a customer’s site, because they had programmed an infinite loop on their event...
View ArticleUS-EAST 1 Outage
Jun 13, 09:44 PDTResolved - Amazon Web Services' US-EAST-1 was offline for around 1.5 hours today. Things were completely out of our hands and pageviews were lost. We were unable to do anything during...
View ArticleDashboard & API Database Migration
Jul 23, 22:07 PDTResolved - Thanks for the patience. Ingest remained online throughout (of course) and the dashboard is now back online. We'll leave a maintenance message up on the dashboard as we copy...
View ArticlePerformance degradation
Oct 23, 07:23 PDTResolved - This incident has been resolved.Oct 23, 07:22 PDTUpdate - And we're back. We're really sorry about the degraded performance we caused and we'll be working on a fix this...
View ArticleDatabase slowdown
Dec 8, 10:32 PSTResolved - And we're done. Thank you for your patience. You should notice that your dashboard is faster now.Dec 8, 10:25 PSTIdentified - We are currently performing a migration of some...
View ArticleDatabase slowdown
Dec 11, 13:30 PSTResolved - This incident has been resolved.Dec 11, 12:28 PSTIdentified - The dashboard will be slow for a little while because we've just launched a new dashboard, along with new...
View ArticleDashboard slowdown following update
Jan 29, 07:49 PSTResolved - Thanks for your patience, the dashboard is now functioning as expectded. For those of you following the updates here, please know that we have plans to improve the stability...
View ArticlePerformance degradation
Apr 4, 08:45 PDTResolved - This incident has been resolved.Apr 4, 08:30 PDTUpdate - Ingest is back online and the application appears to be stable again. We'll continue monitoring. Please accept our...
View ArticleSlowdown and partial outage
Apr 9, 08:00 PDTResolved - We were having issues again despite investing significantly in an upgrade. We've now moved all workloads to a queue system, meaning that data will be put into a backlog &...
View Article
