We can see that Firefox with four content processes fares better than Chrome on all platforms which is reassuring; Chrome is still about 2X worse on Windows and Linux. Our current plan is to only move up to four content processes, so this is great news.
Two content processes is still better than IE, with four we’re a bit worse. This is pretty impressive given last year we were in the same position with one content process.
Surprisingly on Mac Firefox is better than Safari with two content processes, compared with last year where we used 2X the memory with just one process, now we’re on par with four content processes.
I included Firefox with eight content processes to keep us honest. As you can see we actually do pretty well, but I don’t think it’s realistic to ship with that many nor do we currently plan to. We already have or are adding additional processes such as the plugin process for Flash and the GPU process. These need to be taken into consideration when choosing how many content processes to enable and pushing to eight doesn’t give us much breathing room. Making sure we have measurements now is important; it’s good to know where we can improve.
Overall I feel solid about these numbers, especially considering where we were just a year ago. This bodes well for the e10s-multi project.
This is the same setup as last year. I load the first 30 pages of the tp5 page set (a snapshot of Alexa top 100 websites from a few years ago), each in its own tab, with 10 seconds in between loads and 60 seconds of settle time at the end.
Note: There was a minor change to the setup to give each page a unique domain. At least Safari and Chrome are roughly doing process per domain, so just using different ports on localhost was not enough. A simple solution was to modify my /etc/hosts file to add localhost-<1-30> aliases.
Measuring multiprocess browser memory usage is tricky. I’ve settled with a somewhat simple formula of:
Where a parent process is defined as anything that is not a content process (I’ll explain in a moment). Historically there was just one parent process that manages all other processes, this is still somewhat the case but each browser still has other executables they may run in addition to content processes. A content process has a slightly different definition per browser, but is generally “where the pages are loaded” — this is an oversimplification, but it’s good enough for now.
firefox processes launched with the -contentproc command line.
firefox without the -contentproc command line, plugin-process which is used for Flash, etc.
chrome processes launched with the --type command line.
chrome without out the --type command line, nacl_helper, etc.
Safari, SafariServices, SafariHistory, Webkit.Networking, etc.
iexplore.exe process launched with the /prefetch command line.
iexplore without the /prefetch command line.
For Firefox this is a reasonable and fair measurement, for other browsers we might be under counting memory by a bit. For example Edge has a parent executable, MicrosoftEdge.exe, and a different content executable, MicrosoftEdgeCP.exe, arguably we should measure the RSS of one the MicrosoftEdgeCP.exe processes, and USS for the rest, so we’re probably under counting. On the other hand we might end up over counting if the parent and content processes are sharing dynamic libraries. In future measurements I may tweak how we sum the memory, but for now I’d rather possibly under count rather then worry about being unfair to other browsers.
Ubuntu 16.04 LTS
Chrome 54 (see note)
Ubuntu 16.04 LTS
Firefox 55 – 2 CP
Ubuntu 16.04 LTS
Firefox 55 – 4 CP
Ubuntu 16.04 LTS
Firefox 55 – 8 CP
Firefox 55 – 2 CP
Firefox 55 – 4 CP
Firefox 55 – 8 CP
Safari 10.2 (see note)
Edge (see note)
Firefox 55 – 2 CP
Firefox 55 – 4 CP
Firefox 55 – 8 CP
Browser Version Notes
Chrome 54 — aka chrome-unstable — was used on Ubuntu 16.04 LTS as that’s the latest branded version available (rather than Chromium)
Firefox Nightly 55 – 2 CP is Firefox with 2 content processes and one parent process, the default configuration for Nightly.
Firefox Nightly 55 – 4 CP is Firefox with 4 content processes and one parent process, this is a longer term goal.
Firefox Nightly 55 – 8 CP is Firefox with 8 content processes and one parent process, this is aspirational, a good sanity check.
Safari Technology Preview 10.2 release 25 was used on macOS as that’s the latest branded version available (rather than Webkit nightly)
Edge was disqualified because it seemed to bypass the hosts file and wouldn’t load pages from unique domains. I can do measurements so I might revisit this, but it wouldn’t have been a fair comparison as-is.
36 hours from waking to sleeping to get to London. Land at LHR, meet up with most of your crew, take a harrowing drive on the wrong side of the road with a semi-pro rally driver to a town on the outskirts of The City. Start with a full english breakfast, walk the High Street — serious question: does every town have a high street? I’m a fan — hang with our friend Bea for a bit, roll out for dinner, crash on an airbed. Slowly adjust to British accents.
Dinner Surprisingly decent Spanish tapas after a rather long wait. Portland’s more of an under-promise over-deliver place (45 minute wait, really takes 15), London’s definitely an over-promise, under-deliver place (10 minute wait, really takes an hour).
Step two: Travel to The Alps
Two life changing words: airport lounge.
Plan to wake up at 6am, Andrew wakes up earlier, hence we wake up earlier. I am tired. So very tired. Rally drive through what feels like the countryside, Foy remarks the trip is “pretty rural” — definitely charming, there’s even a little pub at fork in the road and nothing around it, this is basically what I’ve always assumed England was like. We’re going to LGW, new airport to check off the list. Today it’s easyJet, very affordable prices, Phil got us “speedy” boarding, I have no idea what that is but I’m assured it means no waiting with the filthy, filthy, masses. Quick trip through security, for some reason my laptop is set aside for enhanced interrogation, scanner dude seems equally confused, shrugs and runs it through again. High fives all around.
Two life changing words: airport lounge. Andrew’s been obsessing about working the travel system, and by god he’s figured it out. He gets the four of us into a lounge for freezies, we grab some solid buffet breakfast, caffeinated beverages, and I’m introduced to the wonderful world of bacon rolls, with HP sauce naturally.
An aside: this is not legit bacon, but what I call sad bacon, or I guess British bacon. It’s kinda hammy, it’s still good, but let’s not pretend it’s bacon. On the flip side, Phil refers to legit bacon as burned bacon. To each their own. Oh also the Brits call legit bacon “streaky bacon” and let’s be honest, that’s pretty darned cute.
Anyhow, decent breakfast, caffeinated, generally relaxed before our flight. Let’s do this. Chill out and read a bit at the gate, but oh yeah we have speedy boarding so off we go. Take that aforementioned filthy masses. Today we learn a bit about easyJet, primarily that it’s mostly awful, in discussions later with Phil I learn it’s basically a crapshoot which in retrospect is probably too generous. Back to speedy boarding: we get to walk down the ramp first to the tarmac, it’s super cold out, and you guessed it, we get to wait on the tarmac, me without my coat on. But we’re the first to wait on the tarmac, you know, instead of the heated tunnel, so yay? Okay we board and we get first row so I guess that’s nice. I’m next to some folk with a newborn, like I dunno, maybe a month old? It’s starts to cry, this does not bode well. Luckily this isn’t my first rodeo: headphones on, Deafheaven playing, Kindle out and I’m good to go. Takeoff is delayed, but they keep the door open so I can see my breath. The flight is uneventful, the baby is chill.
Arrive in Geneva and take a bus from the runway to the airport proper. We have to go through customs. This confuses me to no end, what’s the point of the EU if you have to go through customs. Now a bit of nerding out: Switzerland is, in fact, not in the EU while the UK is (for now at least). On the other hand Switzerland is a Schengen member while the UK is not. Hence customs. But I’m not an EU citizen so I get to slum it through the slow line, which whatever no big deal, because, yes you guessed it, I got a Swiss stamp in my passport, and really that’s what it’s all about. Baggage takes forever and combined with the delayed flight we’ve missed our shuttle, luckily they’ve sent another one. Still heavily jet lagged I grab a stupidly expensive ham sandwich and a can of paprika Pringles — really how I could I not — and off we go to France! Fall asleep for the pretty part of the ride, wake up as we roll up to our rental dusted with snow. This feels like a good omen.
Dinner Stroll downtown Chamonix, we’re pretty jet lagged and choose the first brasserie we can find. The place is pretty nice, I grab a local beer that’s not awful and we all get pizzas. Any pretense of blending in is lost.
Step three: Hit the slopes
A note for folks who have not been to Chamonix before, but maybe another ski resort. Chamonix itself is a sizeable place with a nice downtown area, plenty of shopping, food and whatnot. Very walkable. But this isn’t really a walk to the lift type situation, they have a decent and free bus system running to a bunch of lift areas that are covered by the super-mega-ultimate-pass or whatever the marketing people call it. Our rental is a five minute walk away from the main stop and given my downhill ski boot situation this is a good thing. I wouldn’t recommend being further afield than that, and yes you could technically drive to the slopes but they don’t have much parking and it’s the Alps so there’s an extra terrifying mountain weather factor. Unless you’re a semi-pro rally driver such as Phil just take the bus.
Sunday – Les Houches
I declare mutiny, no more lifts today
Wake up with a massive altitude headache despite copious amounts of water and peeing, find some paracetamol, verify that’s just British for Tylenol. Note to self: don’t drink beer your first day at altitude. Less brain dead folks check the weather, all agree Brévent is where it’s at. The walk over to the bus stop is pretty easy. Our over eager compatriots Phil and Foy — maybe just Phil, sorry Foy — can’t wait for the proper bus so we just take the next one that shows up. Turns out it’s going to Les Houches, which is in fact a ski area, so whatevs. Phil and Foy then bail one stop early — it’s too damned hot in this bus! — Andrew and I throw some bows and make it off the bus as well. Apparently we can catch a gondola here, but as it turns out it’s not really running and people are just standing in a line extending out of the building with sad blank looks. Maybe there’s a strike or something. Hike through a parking lot, slog up three flights of stairs in ski boots to a lift — my companions are snowboarders, it’s not quite so awful for them — Phil’s gone before I get up there — this is a theme — Foy’s accosting the gate, turns out his ticket doesn’t work, heads back down, Andrew and I wait. It’s cold, the snow is wet. Foy heads up, now our tickets don’t work. Le sigh. Slog down to the lift station and are informed our tickets need to be “connected to the internet,” this is such an absurd statement we can’t help but have a good laugh. Back up the stairs. This is the point I realize I’ve been living at sea level too long and want to die. Shit viz is declared by one of the Brits — this will also be a theme — completely overcast, wet snow falling, I learn my pants are not waterproof. The wind gets mean; this is awful. Everyone knows the solution to awfulness is food and warmth: lunch time! Bunch of cute little restaurants nestled around the slopes which is truly how I thought the Alps would be. All are full, quaintly dubbed complet by the French which sounds better than sorry you missed the lunchtime memo, these folks are in for the long haul and you’re SOL. I go ahead and hop in one, it looks promising, Phil says we just want drinks (we do not just want drinks), hostess wanders around a bit, checks in on some folks, walks past an empty table, comes back and informs us very frenchly that “no we cannot.” Next place is closed to us as well. Ski to the bottom to check out the base area. There is no base area to speak of, this is more a place with a lift where the road ends. There are two spots, neither with indoor seating, grumpily eat food outside. I declare mutiny, no more lifts today. Not much complaint, we go home.
Dinner Andrew makes bolognaise.
Monday – Brévent-Flégère
Extreme chills, huddle by the heater. Everything hurts.
Okay Brévent for reals this time. Phil’s still bus adverse so it’s declared we’ll hike up a hill to station, doesn’t look that bad on Google maps! Oh right, it snowed last night which is exciting in the abstract sense, but said hill turns out to be a fun trifecta of snowy, icy, and steep. There isn’t a sidewalk. This ski boot situation is making me consider taking up snowboarding. Anyhow the hike up is no fun, Andrew gets slightly sideswiped by a struggling delivery van. Phil’s disappeared from view, I assume he’s most likely already done a few laps on the mountain. We arrive to somewhat shit viz and experience our first cloudening, which, for the uninitiated, is when you become literally ensconced in a cloud and shit viz becomes near zero viz. Stick it out for a full day, wrap up, drinks at base of the gondola at a self declared slow food bar which somehow doesn’t serve food, but most certainly serves beer. Phil and Foy’s true colors shine as they show us how real Brits drink: which is copious amounts of (admittedly low alcohol) beer along with excellent stories and lots of laughter. I’m starting to feel pretty crappy and lag behind on the drinking, Foy takes pity on me swaps out my untouched pint for his empty glass. Foy you’re a good man. After some shenanigans getting home (impromptu roadside snowboarding, yet another early bus departure) Phil and Foy get deposited at the house. Andrew and I go to the pharmacy and score some cough drops. Head back to gather folks for dinner, alas they’ve retreated to their rooms. Andrew queues up Dirk Gently’s Holistic Detective Agency and I zone out for a while, Phil reappears spry as ever. I swear this guy’s got a fountain of youth hidden somewhere. Collapse in bed for a fever plagued night. Extreme chills, huddle by the heater. Everything hurts. Super hot, lie down on cold tiles for a while, ache, repeat.
Tuesday – Les Grands Montets
I am sick, it’s snowing, this is dumb.
I’m dead to the world. Andrew knowingly comes in my room, hands me some cold pills and cough drops. Get to the mountain okay, a bit snowy. Shit viz, can’t tell where bumps are. I take one run, stare fixedly at the falling snow up higher and decide not to go up to the top with the rest. Pretty loopy on cold medicine I do a few lower elevation runs, eventually topple over comically trying to get to the chairlift. Others come down, declare it the worst ride of the trip. We go home.
That night Andrew and I pop in a pharmacy and after a bit of negotiation get some real drugs. Phil marches ahead and we find him ordering dinner at a place that only has outdoor seating. I am sick, it’s snowing, this is dumb. Finally get real sleep with the help of some sort of French sedative syrup.
Dinner Half a burger on a hard roll.
Wednesday – Nope, nope, nope.
… wake up in the middle of the night drenched in sweat, swig more syrup
Andrew and I give up, stay home. Others go somewhere, more shit viz, come home. My new cold medicine crack pills make me feel somewhat better if not tweaked up, clean the place while Andrew sleeps and insist on using the dishwasher because, well I don’t have one at home, and really it’s the small things. Lounge and read, Foy declares I look like a stoner. This is ok.
Convince the crew to eat late lunch of soup aux pois. Simple food, it does the job. Folks want to wander the town. In wet falling snow. Fuck that. Head home and stop by a legit French butcher, use my memorized phrase to no avail, they have no chicken. Get it from the supermarket like a goddamned animal. I make dinner, Foy declares it Game of Thrones-esque, people seem to like it well enough. This is the best I’ve felt so far, but still pretty shitty. Sleep the sleep of the dead, wake up in the middle of the night drenched in sweat, swig more syrup, go back to sleep.
Dinner Chicken thighs over onions, potatoes, carrots in a red wine reduction.
Thursday – Brévent (again)
We all buy tickets for Italy tomorrow at 9:30. This is optimistic.
Wake up at 9:30 drenched in more sweat. Feel amazing, I guess this is what a fever breaking is like. We realize it’s too late to book tickets to Courmayeur (it’s in Italy, you need to book a bus ticket ahead of time). We take the easy route and go to Brévent, take the fucking bus this time. Somewhat sunny, only a few cloudenings, quite fun. Get to take world’s shortest funicular, day is made. Phil and Foy split off, it’s a bit easier just coordinating two people. Nice fast runs, not too crowded. Sit down lunch of saucisses frites, which turns out is two hotdogs on top of fries. Drink vin chaud for the first time, it’s quite nice. We all buy tickets for Italy tomorrow at 9:30. This is optimistic. My phone dies the final death, screen is completely hosed.
Dinner Raclette at tourist trap (we knew that ahead of time). It’s ridiculous: we’re going to scrape molten gooeyness off half a wheel cheese sitting under a tableside broiler. Phil gets there first and scores us a spot in the “cave.” Foy, Andrew, and I drink a bottle of wine chosen for its characteristic of not being the absolute cheapest. Foy being Foy can’t just have standard raclette, he wants fries too (which to be honest sounds pretty good) and somehow manages to get a steak tartare instead (which happens to come with fries). Luckily he and Phil are fans so no worries. The night goes on and Phil is not well, our plague is now his plague. It’s declared we must get the local specialty, génépy, for desert. It’s some sort of liqueur made of a local flower, I think. Not bad, tastes like sugar free mouthwash. Andrew can’t hang, he chases each sip which a large slug of water. He’s a trooper.
Friday – Courmayeur
I don’t know how young Eric made it down.
Off to Courmayeur, Phil is non-responsive so we leave him to his own devices. Actually go into town for breakfast. Get legit coffee and hit a patisserie for a delicious croissant. Bus ride is interesting, nice views and we take a tunnel through Mt Blanc from France to Italy. The bus stops at sketch completely abandoned parking lot for some skyway thing, we don’t get off. We should have got off. Land in town, it’s a pretty swank place. Andrew pulls up Google maps which yet again directs us up a hill, get passed by posh cars, find a gondola. Foy gets chastised for not having the proper paperwork but we all get in. Oh man the snow is great! Sadly we’re in complete cloud most of the time, worst viz yet. We eat at the Italian ski slope version of a food cart pod. I fumble through ordering in half English, half broken French, a passable grazie and get what I think is going to be vaguely polenta with meat. Yolo. Ends up it’s pretty good. I decide we should take cable car to top because, yay, a slightly new form of transport. This is a bad idea, one run but you can only see maybe 5 feet ahead, this is 100% cloudening, no snow definition whatsoever. I glance back and Andrew’s stopped and is waiting for Foy. Foy eats it because we literally can’t tell which way is downhill, Andrew starts to laugh and then topples over himself. I make it down to next lift area, Foy and Andrew take longer. Foy states: “I don’t know how young Eric made it down”. I’m occasionally referred to as young Eric, I assume this is good. Lower down we have a good time travelling around the mountain, some playing in trees at which point Foy disappears for twenty minutes or so. He’s pretty sure he found some sort of an animal trail. Pause for more drinks and then seek out the mythical gondola to take us back to the bus. We finally find the right spot and head back to the base area on the other side of the highway from the fabled skyway. Scratch our heads a bit and figure out there’s an underpass, get back early and manage to scam our way on to an earlier bus. Go back through the tunnel, and sweet jesus, we’re blinded by sunlight and blue skies on other side. Fuuuuuuuck. Get home, Phil is still dead. Buy a new phone online, pack up, fall asleep early.
Dinner Andrew and Foy indulge me and we go to a place with fondue, they get steaks. I am happy.
Step four: Peace out Alps
easyJet: Life is Pain
I cook everything left in the fridge for breakfast. Leave our leftover beers for next crew. It’s the first blue sky morning, we’re leaving. Chill at the lounge in Geneva, eat some more decent buffet food, free beer, not a bad exit.
We wait out a delay in the lounge and then sprint to the gate when they change the departure time. Roll back to step two, remember easyJet? Yeah they still suck. They load us onto a standing room only bus, drive us over to the plane and then make us stand in the bus for half an hour. Load us up and guess what? Yeah the plane is still delayed but they wanted to load us up “just in case.” Sit on the tarmac for a few hours. I come up with a few new slogans for easyJet as we wait such as: easyJet: Because fuck you, easyJet: Life is Pain, and easyJet: That tea’ll be 2.50. Finally make it to Phil’s place around dinner time, poor Foy still has to drive several hours back home to Leeds.
Dinner Eat out at a curry shop (I’m Britishing so hard right now) with our friend Maggie. I approve.
Step five: Down day in London
We’ve out-Britished ourselves this time.
Wake up and Phil’s already gone, he and Maggie are off to Finland for a couple’s ski weekend. Dude’s a machine. Andrew does some due diligence and arranges a mini-cab for us tomorrow. We take a train (or is it an overground subway?) into London and get off at a station that’s literally on a bridge, more novel transportation for me. We hit up the Tate Modern which is great, use a pedestrian only bridge to cross the Thames, check out a fancy suit store Andrew likes. Take the tube to meet up with Bea again at ginormous comic book store, grab fish and chips (Britishing even harder right now) and have a generally delightful time. Andrew and I attempt to go to the oldest continuously operating pub in London, it’s closed. Go to a pub that’s open instead, it’s clearly better. Get to take a double-decker bus home, top deck, duh, I show some restraint and don’t take the front seat. After dinner we finish packing and get ready for an early morning drive to LHR.
Dinner Grab another pint at the pub that’s been declared reasonably safe, hit up a kebab shop. We’ve out-Britished ourselves this time.
An uneventful drive to LHR, one more lounge and I’m off to Portland again. At this point all that’s left is a lingering cough and a righteous stomach bug that won’t leave me collapsed and shivering on the bathroom floor for a few more days.
The trip was mostly a bust in the skiing and feeling like a human being sense, but it was nice to get out of town, detox from work, and see some good friends. As a rule I’d say don’t bother with Chamonix unless a) you live in Europe and b) you can schedule a trip at the last minute so you know it’ll be decent weather. I might go back in the summer though.
Windows build sadness aside, as I went back to my old standby, Linux, I started to wonder about its build times. Am I doing my absolute best here? Lets do some tests. I’ve been using Clang ever since I need to do asan builds — yeah, yeah I know GCC supports this as of version xyz, but I also get fun things like our static analysis plug-in and pretty error messages.
As an aside: to install Clang I just downloaded the x86_64 Ubuntu build from LLVM’s official site. Nice and easy. For GCC builds go to the gcc site, check out their sweet binaries page where they let you know you can build it yourself you filthy animal and if you’d please download the source from a mirror that’d be great. If we’re evaluating on website alone GCC has already lost.
Why those versions? I was already using clang 3.9.0 (and it was the latest release at the time), gcc 5.4.0 is what ships with Ubuntu 16.04, gcc 6.3.0 is the latest release.
Now you may argue that the resulting executable of foo bar baz is going to be smaller/faster/better. And I’d argue as a dev I don’t care. I just want to build quickly so I can run my test and repro a bug. Generally speaking all my builds are debug, asan-enabled, dmd-enabled. They’re never going to run fast.
I’d love to hear your thoughts on how to get optimal build times of Firefox on all platforms. Maybe I can build with better settings — for GCC I just used a vanilla config I found on another blog. I tried doing a PGO build of clang trained on building Firefox, but I ended up getting worse times with that. Odds are I was doing it wrong, but it would be pretty cool if Mozilla could produce them in our build infrastructure; I filed bug 1326486 for this.
It would be interesting to see the gcc and llvm folks use our codebase as a testing ground for build time performance — maybe it’s time to fire up arewecompiledyet.com.
Recently I was looking into doing more dev on a Windows machine — you know, because basically all our users on are on Windows — and ran into the sad, sad fact that a clean build is going to take me 40 minutes on a brand new “pro-level” laptop. That’s just unacceptable. I filed a bug for that, the response was roughly “yeah we know, get a better machine” and was closed as invalid. I guess I get it, as far as a short term solution that makes sense, but this is pretty sad. It’s sad because on my 3 year old MacBook Pro I get 22 minute clean build for OSX. On my Linux desktop with similar specs I’m seeing 18 minute builds (this is without ccache).
I should note we’re talking about clobber builds here for a platform dev poking around in C++. A lot of work has been done to make builds for frontend folks super fast with artifact builds (we’re talking a minute or two). We could also look at iterative builds, but often I’m working in heavily shared files that it doesn’t really matter (strings etc).
Not all is lost
It sounds like there’s some work to make it better although I don’t have any bug numbers, and have no clue what the priority is. It was also pointed out that sccache is going to work locally on Windows which should be a big improvement, it will be interesting to see some actual numbers.
I’m not sure if there’s more I can do to improve things as-is, here’s what I’ve done so far:
– Disabled malware scanning for my dev directory
– Configured the laptop to “performance” mode
Just piping the build output to /dev/null was actually rather effective, it shaved about 5 minutes from the build time. This isn’t a great solution though as I’d like to see progress and warnings. Another suggestion was to disable parts of Firefox that I don’t need, unfortunately I often tinker with files that are used throughout the codebase so I can’t disable things without worrying about breaking them.
In Firefox we use a custom allocator, mozjemalloc, based on a rather ancient version of jemalloc. The motivation for using a custom allocator is that it potentially gives us both performance and memory wins. I don’t know the full history, so I’ll let someone else write that up. What I do know is that we use it and it behaves a bit differently than system malloc implementations in a rather significant way: minimum alignment.
Why does this matter? Well it turns out C runtime implementations and/or compilers make some assumptions based on what the minimum allocation size and alignment is. For example in bug 1181142 we’re looking at a crash on Windows that happens in strcmp. The CRT decided to walk off the end of a page because it was comparing 4 bytes at a time.
Why was it doing that? Because the minimum allocation size is at least 4-bytes, so why not? If you head over to MSDN it’s spelled out somewhat clearly (although older versions of that page lack the specific byte sizes):
A fundamental alignment is an alignment that’s less than or equal to the largest alignment that’s supported by the implementation without an alignment specification. (In Visual C++, this is the alignment that’s required for a double, or 8 bytes. In code that targets 64-bit platforms, it’s 16 bytes.)
We’ve had similar issues on Linux (and maybe OS X), see bug 691003 for more historical details.
The address of a block returned by malloc or realloc in GNU systems is always a multiple of eight (or sixteen on 64-bit systems).
We haven’t seen a compelling reason to go up to a 8-byte alignment on 32-bit platforms (in the form of crashes) but perhaps that’s due to Linux being such a small percentage of our users.
And lets not forget about OS X, which as far as I can tell has always had a 16-byte alignment minimum. I can’t find where that’s spelled out in bytes, but go bang on malloc and you’ll always get a 16-byte aligned thing. My guess is this is a leftover from the PPC days and altivec. From the malloc man page for OS X:
The allocated memory is aligned such that it can be used for any data type, including AltiVec- and SSE-related types.
Again we haven’t seen crashes pointing to the lack of 16-byte alignment, again perhaps that’s because OS X is also a small percentage of our users. On the other hand maybe this is just an optimization but not an outright requirement.
So what happens when we do the right thing? Odds are less crashes which is good. Maybe more memory usage (you ask for a 1-byte thing on 64-bit Windows you’re going to get a 16-byte thing back), although early testing hasn’t shown a huge impact. Perf-wise there might be a win, with guaranteed minimum sizes we can compare things a bit quicker (4, 8, 16 bytes at a time).
It’s time for a segment I call Storytime with Uncle Eric in which I regale you with tales of woe and triumph from my past and present programming responsibilities.
This is the story of one badass little printer.
To be quite honest, I’m not sure how I got assigned this bug. I was working for a company that made big, as in taller than this writer, optical routers that folks like AT&T and L3 used to run the backbone of the internet. I had interned/contracted there for four years in the testing and automation department — someday I’ll write about that period, it was great — but they wouldn’t hire me until I graduated from college. I’m reasonably sure that’s the only reason I graduated college, so hat tip to them. Once I graduated I managed to snag a permanent position in another department after a particularly awful interview — again another funny story — writing embedded C++ to run on the real-time OS these machines used. So I’m a juniorish SW engineer, but with four years experience at the company. I know how to write Java, C, TCL, Squeak (yeah we learned some real life job skills in college, I swear). It’s cool though, I mean C++ is basically C plus Java right? Oh god, young Eric, you are so precious. Side note: someone finally took pity and handed me Seth Meyer’s Effective C++ and for that I am eternally thankful. I did things like fix random bugs that my boss tossed my way, took over maintenance of our Win32 simulators (these machines were expensive, so simulation was a big deal), expanded our smoke test infrastructure, etc. Cool enough things, but not a ton of work contributing code that ran on the hardware.
And then the bug arrived: Machines were randomly rebooting in one our clients’ labs. That’s bad.
A bit of background: these machines were about the size of a fridge with a bunch of slots, each slot took a rather large network card, aka the line module. In my mind’s eye these things are like 2’X3′, odds are it was a bit smaller, but you get the idea, it was going in the fridge not your desktop PC. Each line module could take several smaller cards that handled different rates of traffic, each of which could be configured to take over for each other if one died. There was a ton of work done on redundancy in our systems, you could set up line modules to switch over to another if one failed, if a fiber was cut you could almost instantaneously reroute, you could set up redundant routing so duplicate data was flowing in case one line went down. Then there was the control module, the heart of the beast, and if that went down the whole system rebooted.
Lets take a moment to imagine a room full of refrigerator-sized-blinky-light-fiber-optic-future-is-now machines whirring like crazy as they reboot. Like hurricane force whirring, huey chopper landing forces. That’s unsettling.
Some work had been done before I got tagged in the bug and they figured out it only happened when this printer was plugged into the lab network. On the plus side there was a solution: don’t plug that printer into the lab network. After this experience, I personally would have just gone Office Space on that printer, but to each their own. On the down side I got tasked with figuring out what the heck was going on, all within the cozy confines of my suburban Atlanta cubicle. For there was a glorious thing called Ethereal, you kids probably call it WireShark now, and someone had recorded the network traffic during one of these glorious events, attached it to a defect report and walked away.
Here’s where I come in. The youngin’. The one that wasn’t frantically writing code for the hot new tech, Gigabit Ethernet. That’s right, we were thinking about that over a decade ago. And that’s how I spent the next week. Staring at a network dump, slowly losing my grip on reality as I became those goddamned packets. Be. The. Packet. Flow like the packet. Ask yourself, am I good packet? Deep in one of these sessions something clicked. What the heck was an address of 0.0.0.0 doing in there. That’s not how IP works. You come from somewhere, packet. Well at least that’s not how it works on a good day. I am a bad packet.
So what kind of shenanigans was this packet getting up to? It was an NTP packet. The printer wanted to know what time it was. It said:
Hi good sirs, might you know the time?
But it didn’t ask one good sir in particular, it asked everyone. And it did it in such a way that it was more like:
Hi good sirs, might you know the time? Also why don’t you contemplate Zeno’s paradox for a while and just go ahead and crash.
Okay, I found a bad packet. But why would we crash?
Some more background: You want to send out a network message, the standard way is to say: gimme a sendin’ thing on address zero. That meant I want to send stuff from this computer, just plug in it’s address as return-to-sender because I’m too lazy figure it out. Well on this unique snowflake of a machine, saying gimme the sendin’ thing on address zero actually plugged in the return address of zero, literally 0, which shouldn’t happen, but it was a only printer and we shouldn’t put such high hopes on them.
That’s my theory at least. I whip up a program in Java that sends a, raw, hand crafted packet — thus getting around the rules that say: “For goodness sake, no, you can’t send a packet with a return address of zero” — punt it at one of our test machines, scamper into the lab to see that bad boy whirring like crazy. Hells yeah, I figured it out. But now what do we do?
As part of our contract with our real-time OS vendor we had the source code. This means I can dig in to the underlying code of the network stack, the piece that was most likely choking on that little rapscallion of a packet. It should be noted I am not a kernel hacker at this point. I’m not even a kernel hacker now, I mean I guess technically I’ve done it once so now I am, but don’t hold me to that. I spelunk through the networking code, it’s about as exciting as you think, and finally find a spot where lo and behold processing a packet with a return address of zero is going to give you a very bad no good time.
So I add a if (0) goto the great trashbin in the sky. Seriously, basically one line. I wrote one line to fix my 2 week journey through madness. Not to worry aspiring programmers! You’ll almost always get something worse: later on in life I got to play the week-staring-at-code-fixed-by-removing-exactly-1-character game. That was super fun!
And then it’s pretty simple. I compile that one file, replace it in our OS library, get it into our core build and move on with life. That’s it. Just walk away.
Hopefully that vendor fixed things along the way, it’s been over a decade, our snapshot of the code was pretty old, so I don’t feel too bad telling the story.
Coda: Now that I think about it, that poor printer must never have known the actual time, tirelessly sending out NTP requests and never getting a response. A miserable, unloved existence. And for that I am thankful.
In my previous post I focused on how Firefox compares against itself with multiple content processes. In this post I’d like to take a look at how Firefox compares to other browsers.
For this task I automated as much as I could, the code is available as the atsy project on github. My goal here is to allow others to repeat my work, point out flaws, push fixes, etc. I’d love for this to be a standardized test for comparing browsers on a fixed set of pages.
As with my previous measurements, I’m going with:
total_memory = RSS(parent) + sum(USS(children))
An aside on the state of WebDriver and my hacky workarounds
When various WebDriver implementations get fixed we can make a cleaner test available. I had a dream of automating the tests across browsers using the WebDriver framework, alas, trying to do anything with tabs and WebDriver across browsers and platforms is a fruitless endeavor. Chrome’s actually the only one I could get somewhat working with WebDriver.
Luckily Chrome and Firefox are completely automated. I had to do some trickery to get Chrome working, filed a bug, doesn’t sound like they’re interested in fixing it. I also had to do some trickery to get Firefox to work (I ended up using our marionette framework directly instead), there aresomebugs, not much traction there either.
IE and Safari are semi-automated, in that I launch a browser for you, you click a button, and then hit enter when it’s done. Safari’s WebDriver extension is completely broken, nobody seems to care. IE’s WebDriver completely failed at tabs (among other things), I’m not sure where to a file a bug for that.
Edge is mostly manual, its WebDriver implementation doesn’t support what I need (yet), but it’s new so I’ll give it a pass. Also you can’t just launch the browser with a file path, so there’s that. Also note I was stuck running it in a VM from modern.ie which was pretty old (they don’t have a newer one). I’d prefer not to do that, but I couldn’t upgrade my Windows 7 machine to 10 because Microsoft, Linux, bootloaders and sadness.
I didn’t test Opera, sorry. It uses blink so hopefully the Chrome coverage is good enough.
The big picture
RSS + USS
Firefox Nightly (e10s)
Google Chrome Unstable
49.0.2618.8 dev (64-bit)
Firefox Nightly (e10s)
46.0a1 20160122030244 (64-bit)
50.0.2631.0 canary (64-bit)
Firefox Nightly (e10s)
47.0a1 20160126030244 (64-bit)
So yeah, Chrome’s using about 2X the memory of Firefox on Windows and Linux. Lets just read that again. That gives us a bit of breathing room.
It needs to be noted that Chrome is essentially doing 1 process per page in this test. In theory it’s configurable and I would have tried limiting its process count, but as far as I can tell they’ve let that feature decay and it no longer works. I should also note that Chrome has it’s own version of memshrink, Project TRIM, so memory usage is an area they’re actively working on.
Safari does creepily well. We could attribute this to close OS integration, but I would guess I’ve missed some processes. If you take it at face value, Safari is using 1/3 the memory of Chrome, 1/2 the memory of Firefox. Even if I’m miscounting, I’d guess they still outperform both browsers.
IE was actually on par with Firefox which I found impressive. Edge is using about 50% more memory than IE, but I wouldn’t read too much into that as I’m comparing running IE on Windows 7 to Edge on an outdated Windows 10 VM.
With the e10s project full steam ahead, likely to be enabled for many users in mid-2016, it seemed like a good time to measure the memory overhead of switching Firefox from a single-process architecture to a multi-process architecture. The concern here is simple: the more processes we have, the more memory we use. Starting Q4-2015 I began setting up a test to measure the memory usage of Firefox with a variable amount of content processes.
For the test I used a slightly modified version of the AWSY framework that I maintain for areweslimyet.com. This test runs through a sample pageset, the same one used in Talos perf testing, in an attempt to simulate a long-lived session.
Open Firefox configured to use N content processes.
Measure memory usage.
Open 100 urls in 30 tabs, cycling through tabs once 30 are opened. Wait 10 seconds per tab.
Measure memory usage.
Close all tabs.
Measure memory usage.
For this test I performed two iterations of this, reporting the startup memory usage from the first and the end of test memory usage (TabsOpen, TabsClosed) for the second.
Note: Just summing the total memory usage of each Firefox process is not a useful metric as it will include memory shared between the main process and the content processes. For a more realistic baseline I chose to use a combination of RSS and USS (aka unique set size, private working bytes):
Settled: 30 seconds have passed since previous checkpoint.
ForceGC: We manually invoked garbage collection.
We list the memory usage for each checkpoint using 0, 1, 2, 4, 8 content processes.
Windows 7, 64-bit
Simply put: the more content processes we use, the more memory we use. On the plus side it’s not a 1:1 factor, with 8 content processes we see roughly a doubling of memory usage on the TabsOpenSettled measurment. It’s a bit worse on Windows, a bit better on OSX, but it’s not 8 times worse.
Overall we see a 10-20% increase in memory usage for the 1 content process case (which is what we plan on shipping initially). This seems like a fair tradeoff for potential security and performance benefits, but as we try to grow the number of content processes we’ll need to take another look at where that memory is being used.
For the next steps I’d like to take a look at how our memory usage compares to other browsers. Expect a follow up post on that shortly.