Troubleshooting a MongoDB Performance Issue

UPDATE (2018-06-28): I actually sent a link to this article to the author of the previous blog post and in her reply she indicates that the improvements to cache management and checkpoint areas were more likely to have improved my situation. Just wanted to call out how approachable the MongoDB team is even with these one-off type issues :). Thanks Sue!

UPDATE (2018-06-21): As we were running MongoDB 3.0.15 while all these issues were going on it’s entirely possible that the optimizations made to the write-ahead log of WiredTiger may have also contributed to this improvement in performance :)

The following is an edited excerpt from an email I sent out internally about an intermittent performance issue we’ve been experiencing for several years now. The daily processing challenges we’ve been experiencing revolved around running server-side javascript in order to produce daily reports. As our data ingestion rates rose and our data processing needs climbed, our server performance continued to degrade. This would occur regardless of the size of the VMs we would spin up.


Our MongoDB cluster is configured with three (3) servers: 1x primary (write-enabled) and 2x secondaries (read-only). These are running on Azure DS14v2 VMs with 8TB of storage (8x 1TB striped via LVM as these were the largest premium SSD-based data disks available at the time).

Aside from the servers being scaled up periodically, this configuration has been constant since the inception of the product.

The only major upgrade came in the form of a migration from 2.6 to 3.0 in 2015. At the time this was a major shift as it required rewriting a number of the underlying system scripts as well as introducing LRS-based storage to try and squeeze some additional performance out of the disks. Why optimize for IOPS? Because the reporting platform was designed to copy a lot of data back and forth in order to generate reports segmented by dimension (“Group”, “Company”, “Country”, “State”, “City”).

This chart (48 hours sampled from 1 week ago) shows Cache Usage spiking and Replication Lag spiking. The cache spikes occur as new writes trigger index activity, which invalidates (dirties) cached memory and causes cache eviction.

Read on →

Setting up domain forwarding in

Last Updated:

I’ve known for a long time that when you navigate to my domain directly at that you would be redirected to a Hover placeholder page.

I’ve meant to add a domain redirect for a long time but just never got around to it … until now.

If you log into your Hover control console at<your domain>, you can just add the forward from the Mangage Forwards section.

  • Click Create a Forward
  • Select from the dropdown list
  • Enter the full url (http://...) you would like requests to your domain to go to
  • Click Save Forward

After about 15 minutes this will be active and all requests to your domain will redirect to the url you’ve selected.

Redmine Plugin Extension and Development Still In Demand

Last Updated:

It’s been a while since I last wrote about Redmine Plugin Extension and Development so I thought I’d give a quick update.

I have been sharing sales numbers whenever possible as a way of (hopefully) encouraging other authors to see that there is some money out there, even for obscure niche topics.

Ebook Print
Q2/2017 17 12
Q1/2017 16 8

This isn’t a super lucrative endeavour, but considering I published this book three years ago and it’s still in demand, I can’t complain.

If I had more time on my hands I might look into writing about something a bit more “in demand”, like Docker or .NET Core (random topics I’m currently interested in :P).

Turning an old Android Phone into a Plex Media Server + PVR

This is possibly a solution to a problem no one other than me has, but once in a while I like to challenge myself to see if something ridiculous is possible.

This time around I wanted to see if I could take my old Moto X (2014) and use it as a Plex Media Server.

I didn’t want to just see if I could install Plex though; I wanted to see if it would be possible to actually run a PMS instance along with Sonarr to download content automatically for shows I’m interested in.

This introduced a couple of challenges, as Sonarr would need to be run using Mono (as it’s a .NET project), I’d need a Bittorrent client to actually download the content, and I’d likely need to be running a Jackett service to allow Sonarr to process request through sites like The Pirate Bay.

Finally, as the Moto X is an ARM device, all of our software will need to be capable of running on an ARM platform.

Note that this process (with some minor changes) can be used to get a Plex + Sonarr + Jackett + Filebot + Transmission setup done on any Linux distribution. I just thought it would be fun to do it using a phone.

Read on →

Just Finished - Warcraft Adventures

Although I’m a huge fan of the point-and-click adventure genre, I’ve been primarily focusing on covering 16-bit RPGs in my spare time. I’ve still got a sizeable backlog to wade through, but when the news dropped that Warcraft Adventures: Lord of the Clans had been leaked, I couldn’t resist.

Although no one really got a taste of this game until 2010 when MAN-biker posted some content to Youtube, I’d been inquiring with Blizzard as early as 2008. I still have a copy of the email, so I’ve trimmed it down a bit and shared it below.

Read on →

Summer RPG Update 2016

I don’t really have much progress to report for the last month, so I thought I’d throw out a few of the games I’ve been working through to see if I can generate any interest in the next article ;)

First off, I’ve been playing through Golden Sun for the GBA.

I really like the psyenergy system and how you use it for puzzle solving. It makes this game a bit more of an ARPG along the same lines as Lufia 2.

I’ve made it as far as Kraken, but I’m finding beating him is proving to be more difficult that I’d expected. Probably going to have to back out and grind for a while, which I don’t have a lot of time for …

Next, I’m pretty sure I’ve given up on the 7th Saga. I’m still at Telaine with a party at level 17, but I’m pretty sure that to progress, there is grinding in my future.

Similar to the situation I’m having above with Golden Sun, i just need to set many hours aside to level up in order to proceed, but with the amount of time I’ve got these days to devote to these games, that may never happen.

Read on →

Just Finished - Dragon Quest I

I actually finished this game around the end of June, but haven’t really had a chance to write about it until now. I’m currently sitting out by the lake at the cottage and figured it was about time :P

The last time I played through Dragon Quest was likely in the late 80’s. I’m pretty sure I either heard about it, or got a copy of it through Nintendo Power, and like most kids in North America at the time, this would have been my first introduction to what would become known as JRPGs.

I decided to go with the SNES re-release of this title for my playthrough as I wanted some updated graphics and music. Although the SNES version was never officially localized for North America, there is an excellent fan translation available.

It threw me off a bit when they refer to the legendary hero as Roto (as opposed to Erdrick), but I’m assuming this is a better translation than what we got in the 80’s so that’s probably what the name should have been in the first place.

Dragon Quest was essentially the first JRPG. As a result, there are a lot of concepts introduced here that would be adapted and refined by other titles and series over the years. This means that some parts of this game feel a bit rough or unbalanced.

I originally played the NES version of this title, and have fond memories of just how brutally hard it was. This game introduced me to what I would come to know as “grinding”, as you couldn’t progress through the game unless your character was sufficiently leveled up in order to tackle the monsters in the areas you were exploring.

This meant walking back and forth and fighting random monsters.

Enemy encounter rates are high. Very high. I think the SNES version actually optimized this a bit, but the rates are still high. This is useful for grinding, but gets tedious when you want to explore, or really need to get back to a town to heal.

The story is pretty simple.

You’re the descendant of the legendary hero, and have been tasked with rescuing the princess and defeating the Dragon Lord.

Read on →

Just Gave Up On - Shadowrun (Genesis)

UPDATE: 2016-09-05 - It was pointed out on the Reddit thread and in the comments below that I might have used a cheat which prevented finishing the game. I didn’t think I’d used one, but I think while doing research for the article i may have been testing it out :(

Note that you don’t really need to cheat in this game to get more Nuyen; just grind ;)

Thanks to those who responded. It’s worth noting that if you’re going to play this game, avoid the cheat menu as it will bite you in the ass.

As I work my way through the 8-bit and 16-bit games I grew up with, there are (and will continue to be) those that I just can’t get through. Shadowrun for the Sega Genesis turned out to be one of those titles.

I’ve made it all the way to the final boss, and though he doesn’t beat me, I empty all my clips (and my runner’s clips) into him and just can’t seem to kill him and trigger the endgame sequence.

This is extremely frustrating as I’ve effectively finished this game, but if I can’t beat the last boss I can’t in good conscience call this a “Just Finished” article.

Regardless, I’ve gone through enough of this game to be able to write about it, and based on my experience, recommend it to other players looking for some good retro cyberpunk action-RPG gaming.

Read on →

Extracting Best ROM from GoodTools Generated ROM Sets

Last Updated:

As a kid of the 80’s, I have fond memories of all the old 8-bit and 16-bit consoles that I grew up with.

Although it’s easy enough to find ROMs, I tend to find myself going for the GoodTools generated sets more often than not as they’re considered “complete”.

This is kind of ridiculous as I don’t speak Japanese, which constitutes the vast majority of the contents of these sets.

Even though most emulators support compressed ROM sets, I’d prefer to just have the English ROMs available on their own in one place.

As a programmer, I thought “How can I do this in Linux?”, but more specifically, “how do I do this from the command line directly?”.

# extract best rom to directory
# best contains !
7z e "*.7z" -o../../ *[!]*.* -r

# purge all non US/European
find . -type f ! -name '*(U)*' ! -name '*(E)*' -delete

# purge duplicates where a (U) exists alongiside an (E)
for f in *"(E)"*; do us=`echo $f | sed -r 's/\(E\)+/\(U\)/g'`; if [ -e "$us" ]; then echo "FOUND $us - removing $f"; rm "$f"; fi; done

If you find yourself with compressed ROM sets and you want to just grab the English ones, this might just come in handy ;)

Redmine Knowledgebase 3.2.0 Released

I haven’t been very actively involved with this plugin or the Redmine community as a whole lately, but it would seem there is a very active user-base still logging bugs and enhancing this project.

You can grab a copy of the release on GitHub.

I’m pushing out version 3.2.0 of the plugin thanks to the efforts of some very dedicated community members, who I’d like to highlight below:

Thanks to Frederico Camara:

  • updating acts_as_rated to work with Redmine 3.2.x

Thanks to Eduard Kuleshov:

  • getting this plugin supported in Redmine 3.0.x

Thanks to Axel Kämpfe:

  • getting this plugin supported in Redmine 3.1.x and 3.2.x

HUGE thanks to Rob Spearman for basically taking over the project and pushing it forward:

New Configuaration options

  • Show articles without tabs
  • Show attachments before article content
  • Show thumbnails for articles in lists
  • Show breadcrumbs for articles in lists

New permissions

  • Article history will only show up if have view permission
  • optional permission for users to manage just their own articles. (#306)


  • Sort Tags on the index page
  • Added authored view so users can find articles by author easily
Read on →