Technical Services Engineering at MongoDB

The goal of this post is to provide a first hand account of what it means to be a Technical Services Engineer at MongoDB, as well as what the journey getting to this point has looked like for me.


I have been working in Application Development and Software Engineering for nearly two decades. I started off writing desktop applications in QuickBASIC and Turbo Pascal, then eventually in VB6, VB.NET, C++ and C#. When it was time to shift focus to web development I started off with HTML/JS/CSS (as we all do :P), then in Flash/AS3, Flex, Python, Ruby/Rails and Node.js.

I have been writing software since I was a kid, starting with some automation tools for my mom’s business. I then moved on to building tools to help me cheat at various games I was playing at the time, and eventually got more into emulator programming and reverse engineering. I guess you could say I’ve always loved solving problems programmatically, and especially enjoyed identifying opportunities for automation and custom tooling.

This led me down an informal DevOps track, as I was finding there was a need for optimization in the infrastructure layers that my applications were deployed to. This led me deeper into Linux internals, system administration and network operations.

While I was gaining these new skill-sets my primary focus was always on application development and delivery. Before coming to MongoDB I was working as a Development Lead / System Architect, but I found that my focus was always being drawn back to solving performance challenges at the infrastructure level.

Read on →

Hello MongoDB

Last Updated: Jan 28, 2015

As of August 13th, I am no longer a System Architect at DAC Group. I have a public post on LinkedIn that got some good traction, but to summarize it was time to move on.

I’ve been a software engineer in some capacity or another for nearly 20 years now. The position I’ve taken is as a Technical Services Engineer, which is more of a support role than an active development role.

The decision to make this move wasn’t make lightly. I’ve been working hands on with code or overseeing a team of developers on a day to day basis for most of my professional career. As such, I was also involved with software engineering, and this was no different in my role as a System Architect.

In that role, I was still committing code on a nearly daily basis. If not, I was performing code review, or working on a design for a new system or solution. I would consider this all to still be “hands on”, though I had found myself mired in DevOps work a lot more than I would have liked (there were not sufficient Linux Sysadmins available to assist with the type of server operations oversight that was required).

The role at MongoDB isn’t a traditional “Tech Support” type of role, as it requires a strong knowledge of networking, databases, system design, programming and client services. I’ve been a fan of the MongoDB server for over 8 years now, and have brought it along with me to several new consulting opportunities as well as the full time jobs I’ve help. I believe very strongly in the quality of this product, as well as the peripheral products that they’ve developed.

I think the time has come for a new adventure. This is the first step towards a new career journey with a new company, as opposed to an incremental move upwards within the same professional space.

Troubleshooting a MongoDB Performance Issue

UPDATE (2018-06-28): I actually sent a link to this article to the author of the previous blog post and in her reply she indicates that the improvements to cache management and checkpoint areas were more likely to have improved my situation. Just wanted to call out how approachable the MongoDB team is even with these one-off type issues :). Thanks Sue!

UPDATE (2018-06-21): As we were running MongoDB 3.0.15 while all these issues were going on it’s entirely possible that the optimizations made to the write-ahead log of WiredTiger may have also contributed to this improvement in performance :)

The following is an edited excerpt from an email I sent out internally about an intermittent performance issue we’ve been experiencing for several years now. The daily processing challenges we’ve been experiencing revolved around running server-side javascript in order to produce daily reports. As our data ingestion rates rose and our data processing needs climbed, our server performance continued to degrade. This would occur regardless of the size of the VMs we would spin up.


Our MongoDB cluster is configured with three (3) servers: 1x primary (write-enabled) and 2x secondaries (read-only). These are running on Azure DS14v2 VMs with 8TB of storage (8x 1TB striped via LVM as these were the largest premium SSD-based data disks available at the time).

Aside from the servers being scaled up periodically, this configuration has been constant since the inception of the product.

The only major upgrade came in the form of a migration from 2.6 to 3.0 in 2015. At the time this was a major shift as it required rewriting a number of the underlying system scripts as well as introducing LRS-based storage to try and squeeze some additional performance out of the disks. Why optimize for IOPS? Because the reporting platform was designed to copy a lot of data back and forth in order to generate reports segmented by dimension (“Group”, “Company”, “Country”, “State”, “City”).

This chart (48 hours sampled from 1 week ago) shows Cache Usage spiking and Replication Lag spiking. The cache spikes occur as new writes trigger index activity, which invalidates (dirties) cached memory and causes cache eviction.

Read on →

Setting up domain forwarding in

Last Updated: Jan 28, 2015

I’ve known for a long time that when you navigate to my domain directly at that you would be redirected to a Hover placeholder page.

I’ve meant to add a domain redirect for a long time but just never got around to it … until now.

If you log into your Hover control console at<your domain>, you can just add the forward from the Mangage Forwards section.

  • Click Create a Forward
  • Select from the dropdown list
  • Enter the full url (http://...) you would like requests to your domain to go to
  • Click Save Forward

After about 15 minutes this will be active and all requests to your domain will redirect to the url you’ve selected.

Redmine Plugin Extension and Development Still In Demand

Last Updated: Jan 28, 2015

It’s been a while since I last wrote about Redmine Plugin Extension and Development so I thought I’d give a quick update.

I have been sharing sales numbers whenever possible as a way of (hopefully) encouraging other authors to see that there is some money out there, even for obscure niche topics.

Ebook Print
Q2/2017 17 12
Q1/2017 16 8

This isn’t a super lucrative endeavour, but considering I published this book three years ago and it’s still in demand, I can’t complain.

If I had more time on my hands I might look into writing about something a bit more “in demand”, like Docker or .NET Core (random topics I’m currently interested in :P).

Turning an old Android Phone into a Plex Media Server + PVR

This is possibly a solution to a problem no one other than me has, but once in a while I like to challenge myself to see if something ridiculous is possible.

This time around I wanted to see if I could take my old Moto X (2014) and use it as a Plex Media Server.

I didn’t want to just see if I could install Plex though; I wanted to see if it would be possible to actually run a PMS instance along with Sonarr to download content automatically for shows I’m interested in.

This introduced a couple of challenges, as Sonarr would need to be run using Mono (as it’s a .NET project), I’d need a Bittorrent client to actually download the content, and I’d likely need to be running a Jackett service to allow Sonarr to process request through sites like The Pirate Bay.

Finally, as the Moto X is an ARM device, all of our software will need to be capable of running on an ARM platform.

Note that this process (with some minor changes) can be used to get a Plex + Sonarr + Jackett + Filebot + Transmission setup done on any Linux distribution. I just thought it would be fun to do it using a phone.

Read on →

Just Finished - Warcraft Adventures

Although I’m a huge fan of the point-and-click adventure genre, I’ve been primarily focusing on covering 16-bit RPGs in my spare time. I’ve still got a sizeable backlog to wade through, but when the news dropped that Warcraft Adventures: Lord of the Clans had been leaked, I couldn’t resist.

Although no one really got a taste of this game until 2010 when MAN-biker posted some content to Youtube, I’d been inquiring with Blizzard as early as 2008. I still have a copy of the email, so I’ve trimmed it down a bit and shared it below.

Read on →

Summer RPG Update 2016

I don’t really have much progress to report for the last month, so I thought I’d throw out a few of the games I’ve been working through to see if I can generate any interest in the next article ;)

First off, I’ve been playing through Golden Sun for the GBA.

I really like the psyenergy system and how you use it for puzzle solving. It makes this game a bit more of an ARPG along the same lines as Lufia 2.

I’ve made it as far as Kraken, but I’m finding beating him is proving to be more difficult that I’d expected. Probably going to have to back out and grind for a while, which I don’t have a lot of time for …

Next, I’m pretty sure I’ve given up on the 7th Saga. I’m still at Telaine with a party at level 17, but I’m pretty sure that to progress, there is grinding in my future.

Similar to the situation I’m having above with Golden Sun, i just need to set many hours aside to level up in order to proceed, but with the amount of time I’ve got these days to devote to these games, that may never happen.

Read on →

Just Finished - Dragon Quest I

I actually finished this game around the end of June, but haven’t really had a chance to write about it until now. I’m currently sitting out by the lake at the cottage and figured it was about time :P

The last time I played through Dragon Quest was likely in the late 80’s. I’m pretty sure I either heard about it, or got a copy of it through Nintendo Power, and like most kids in North America at the time, this would have been my first introduction to what would become known as JRPGs.

I decided to go with the SNES re-release of this title for my playthrough as I wanted some updated graphics and music. Although the SNES version was never officially localized for North America, there is an excellent fan translation available.

It threw me off a bit when they refer to the legendary hero as Roto (as opposed to Erdrick), but I’m assuming this is a better translation than what we got in the 80’s so that’s probably what the name should have been in the first place.

Dragon Quest was essentially the first JRPG. As a result, there are a lot of concepts introduced here that would be adapted and refined by other titles and series over the years. This means that some parts of this game feel a bit rough or unbalanced.

I originally played the NES version of this title, and have fond memories of just how brutally hard it was. This game introduced me to what I would come to know as “grinding”, as you couldn’t progress through the game unless your character was sufficiently leveled up in order to tackle the monsters in the areas you were exploring.

This meant walking back and forth and fighting random monsters.

Enemy encounter rates are high. Very high. I think the SNES version actually optimized this a bit, but the rates are still high. This is useful for grinding, but gets tedious when you want to explore, or really need to get back to a town to heal.

The story is pretty simple.

You’re the descendant of the legendary hero, and have been tasked with rescuing the princess and defeating the Dragon Lord.

Read on →

Just Gave Up On - Shadowrun (Genesis)

UPDATE: 2016-09-05 - It was pointed out on the Reddit thread and in the comments below that I might have used a cheat which prevented finishing the game. I didn’t think I’d used one, but I think while doing research for the article i may have been testing it out :(

Note that you don’t really need to cheat in this game to get more Nuyen; just grind ;)

Thanks to those who responded. It’s worth noting that if you’re going to play this game, avoid the cheat menu as it will bite you in the ass.

As I work my way through the 8-bit and 16-bit games I grew up with, there are (and will continue to be) those that I just can’t get through. Shadowrun for the Sega Genesis turned out to be one of those titles.

I’ve made it all the way to the final boss, and though he doesn’t beat me, I empty all my clips (and my runner’s clips) into him and just can’t seem to kill him and trigger the endgame sequence.

This is extremely frustrating as I’ve effectively finished this game, but if I can’t beat the last boss I can’t in good conscience call this a “Just Finished” article.

Regardless, I’ve gone through enough of this game to be able to write about it, and based on my experience, recommend it to other players looking for some good retro cyberpunk action-RPG gaming.

Read on →