Singularity, Part IV: Performance Testing

Articles in this series
Singularity, Part I: Confessions of a Digital Pack Rat
Singularity, Part II: The Network Upgrade
Singularity, Part III: Construction
Singularity, Part IV: Performance Testing
Singularity, Part V: Conclusion

Intro

With Sin successfully built the big question on our minds was how ZFS would compare to ext3 + md software raid on our hardware. To discover this we spent the first weekend after the server was built running the benchmarks below.

Setup

Unfortunately Ubuntu out of the box had problems with the two array drives sitting on the motherboard rather than the PCI-X controller. We found a bug report for it but didn't feel like updating the kernel just to fix it. Instead we built an 8-disk array instead of a 10-disk for these tests. We're now running a full 10-disk array which obviously performs slightly better than the 8-disk benchmarks below.

What impressed me most about ZFS is that it doesn't need to construct the new disk array. On Linux, md must construct the array, usually by flagging one of the disks as faulty and rebuilding. For both the raid5 and raid6 setups below this took over nine hours under Ubuntu. ZFS, by contrast, doesn't (at least as long as you give it entire disks to use). The array was available for us immediately.

Our tests involved unpacking (not uncompressing) the Linux 2.6.23 kernel tarball, writing a 1GB dd file filled with zeros, and using a benchmarking utility called Bonnie++.

Click here for the rest of this article.

This is an oldie, but a goodie.

I guess some parents have trouble getting their kids to come for the holidays

<3 Bash

Singularity, Part III: Construction

Articles in this series
Singularity, Part I: Confessions of a Digital Pack Rat
Singularity, Part II: The Network Upgrade
Singularity, Part III: Construction
Singularity, Part IV: Performance Testing
Singularity, Part V: Conclusion

Intro

This part has been delayed a bit. First, at least one memory DIMM was bad. Second, the motherboard had two fried SATA ports. Thrid, it was Thanksgiving weekend, so getting RMAs placed was nearly impossible.

Still, we got everything up and running. I won't do much more talking here. This part is mainly for the pics! ;)

Click here for the rest of this article.

Singularity, Part II: The Network Upgrade

Articles in this series
Singularity, Part I: Confessions of a Digital Pack Rat
Singularity, Part II: The Network Upgrade [Image 1] [Image 2]
Singularity, Part III: Construction
Singularity, Part IV: Performance Testing
Singularity, Part V: Conclusion

Home Gigabit Network

Gigabit networking has recently become an option for consumer budgets. This advance has been fueled by the proliferation of cheap, unmanaged switches by Netgear and Cisco.

Most of these were four-port or possibly eight-port creatures, however--most consumers didn't have a need for more than this. Businesses still preferred more expensive managed switches for their production network, and their office network often didn't need it. Especially if everyone is in love with wireless.

This is changing, slowly, as people become disillusioned with cubical wireless and trend back toward wired access. As storage continues to increase the uses for it explode. More and more people start to demand gigabit speeds.

Many corporations have already deployed gigabit managed switches for their office environments. Many small businesses, however, have not. Larger-capacity unmanaged gigabit switches cater to small businesses like these, as well as the enthusiast market.

The Gigabit Myth

People often run network tests and find they're not getting "true" gigabit speeds (128 MiB/s). Typically they blame the switch. To some extent they're correct--cheap, unmanaged switches rarely perform well in a single transfer vs. their pricier cousins. There are other reasons, however.

Even with a RaptorX SATAII drive, the absolute best-case transfer rate you're going to get (buffer -> disk) is 84 MB/s. I'm not sure if this is MB or MiB--regardless it's a far cry from 128MiB/s. And that's just your raw 10k disk speed. If your network ports are built into your motherboard you're going to have a slowdown any time your CPU is busy (like when it's buffering and writing disk data). PCI network cards are typically no better, unless it's a PCI-X or PCI-E with offloaded processing.

Where gigabit networks shine, however, is in multiple transfers. Sure, any single given device may not seem like it's taking advantage of gigabit--but when two or more devices are transferring data at the same time you will definitely notice a difference.

The New Network

We jumped to gigabit for the first time when we built Sol. We quickly ran out of ports, however, and have been hobbling along with a 16-port 10/100 Linksys trunked switch ever since.

With cheaper, higher-capacity gigabit switches now available we took this opportunity to scoop one up.


We also had a cabling rat's nest problem, as most people probably do. You never have the right length of ethernet cables so inevitably end up with 15ft monsters connecting devices which are two feet from each other. We decided we wanted to buy our own cabling kit and crimp new cables ourselves so we could better control the length. We found a good deal on a kit:


We especially enjoyed the Engrish instructions :)


Beyond Gigabit

Now that we had an excess of ports it was time to push Sol beyond gigabit speeds. Since we had two gigabit NICs available here we decided to aggregate them into a single, bonded device.

Network bonding is a method of combining two independent NICs into a single virtual device to provide both redundancy and higher throughput. Under Linux there are many different network bonding modes available, from simple round-robin load balancing to advanced load balancing. For our purposes we chose mode 6, or "Adaptive load balancing". The actual details of network bonding is beyond the scope of this document; more information can be found in your kernel distribution (/usr/src/linux/Documentation/networking/bonding.txt)

Now our current fileserver is running at two gigabit while all our other nodes are at regular gigabit. We are now I/O bound on Sol, however this should speed up simultaneous access. As an added bonus, we can run new ethernet cables and swap ports without causing a network interruption.

Next: Singularity, Part III: Construction

Singularity, Part I: Confessions of a Digital Pack Rat

Articles in this series
Singularity, Part I: Confessions of a Digital Pack Rat
Singularity, Part II: The Network Upgrade [Image 1] [Image 2]
Singularity, Part III: Construction
Singularity, Part IV: Performance Testing
Singularity, Part V: Conclusion


Prologue

My desk is many men's worst nightmare. Papers are strewn everywhere. Pencils pour from every crevice. Network charts which should be on the wall are buried under stacks of paper and cdroms. That paperweight given to me as a gift is probably at the bottom of my trash bin--I trash anything that's not absolutely needed. If I didn't I'd drown.

It's not dirty, you understand. It's cluttered. Disorganized.

It's a stark contrast to my digital world. Here I'm a meticulously-organized pack rat. I keep everything I download--I probably still have the NVidia drivers for my original GForce card somewhere under my vast hierarchical construct.

Such an addiction comes at a cost. My habit quickly outpaced available storage. Luckily for me we live in the future, where gigantic fileservers can be assembled from consumer parts. Where a digital pack rat can thrive.

This is the story of the singularity--the fileserver that will succeed sol and retire jupiter. The third incarnation of primal consumer storage.

Click here to view the full article.

Request for sick leave
Toronto Star newspaper.

One hour after beginning a new job which involved moving a pile of bricks from the top of a two-story house to the ground, a construction worker in the town of Peterborough, Ontario, suffered an accident which hospitalized him. He was instructed by his employer to fill out an accident report. It read:

"Thinking I could save time, I rigged a beam with a pulley at the top of the house, and a rope leading to the ground. I tied an empty barrel on one end of the rope, pulled it to the top of the house, and then fastened the other end of the rope to a tree. Going up to the top of the house, I filled the barrel with bricks.

"Then I went down and unfastened the rope to let the barrel down. But, unfortunately, the barrel of bricks was now heavier than I, and before I knew what was happening the barrel jerked me up in the air.

"I hung on to the rope, and halfway up I met the barrel coming down, receiving a severe blow on the left shoulder.

"I then continued to be pulled to the top, banging my head on the beam and then jamming my fingeres into the pulley.

"When the barrel hit the ground, the bottom burst, spilling the bricks. As I was now heavier than the barel, I started down at high speed.

"Halfway down I met the empty barrel as it was coming up, receiving several cuts from the sharp edges of the barrel boards.

"At this point I must have become confused, because I let go of the rope. The barrel came down, striking me on the head, and I woke up in the hospital.

"I respectfully request a day of sick leave."

A Captain's Tale

Long ago lived a seaman named Captain Bravo. He was a manly-man who showed no fear in facing his enemies. One day, while sailing the seven seas, a look-out spotted a pirate ship and the crew became frantic. Captain Bravo bellowed "Bring me my Red Shirt." The First Mate quickly retrieved the captain's red shirt and whilst wearing the bright frock the captain led his mates into battle and defeated the pirates.

Later on, the look-out again spotted not one, but two pirate ships. The captain again howled for his red shirt and once again vanquished the pirates. That evening, all the men sat around on the deck recounting the day's triumphs and one of the them asked the captain: "Sir, why did you call for your red shirt before battle?" The captain replied: "If I am wounded in the attack, the shirt will not show the wound and thus, you men will continue to resist, unafraid." All of the men sat in silence and marveled at the courage of such a manly man.

As dawn came the next morning, the look-out once again spotted not one, not two, but TEN pirates ships approaching. The rank and file all stared at the captain and waited for his usual reply. Captain Bravo calmly shouted: "Get me my brown pants."

I don't like cell phones. I have one, you understand--I just don't like it. I don't like talking to people on the phone. I don't like being left voicemail. I don't like carrying the thing around in my pocket. And I especially don't like when it decides to ring.

To be fair, my cell phone doesn't like me, either. Forced together by fate, we have a familiar if uncomfortable relationship. Like my pager (which I refuse to combine with my cell phone in the form of SMS messages) I'm all but required to have one for my job. And as much as I hate my cell phone, I hate my home phone ringing even more.

So I have one. It sits in my pocket, contemplating its existence as a clock rather than a communications device. Alone and overcharged.

In a way I had that in common with my palm-sized friend. We were both overcharged. I might not have a cable plugged up my ass for a third of my day but I certainly felt like I did each time I payed the bill. I mean, seriously, I use my phone every other blue moon. Why the fuck was I paying so much?

Cheap cell phone plans just don't exist. They all bury you under hundreds or thousands of minutes and free airtime. You have friends, don't you? You're not a loser, are you? If not, you need four thousand anytime minutes!

Then I remembered the new pre-paid phone plans they were marketing to poor people. Ah, I thought. That's the answer! That's my Salvation (Army)!

So began my two-week ordeal.

After shopping around a bit I decided on an AT&T Go-Phone. They offer two plans: one for idiots ($0.10 / minute, + $1 day for each day you use it) and one for people who could perform basic math ($0.25 / minute). I'll let the astute reader determine which one I chose.

There I was using an on-line interface to buy a pre-paid cell phone whose intended market would never be on-line in the first place. AT&T's system was obviously designed for more traditional plans, but I made it work. Somehow. I requested my old Verizon order be ported, bought the cheapest compatible phone and hit submit.

Then waited. And waited. And waited.

After half a week I became concerned and called AT&T's phone support. Amazed that anyone had even attempted to buy a pre-paid phone on-line I was transferred from department to department until someone was finally able to look up my confirmation request. After being escalated up the tree hierarchy to a level of support endowed with actual gray matter I was assured my order would be processed the next day.

And it was. My phone was on its way!

Last Friday it arrived. All that was left was left was to insert the battery and activate the damned thing. Oh, if it had been so easy.

Someone, somewhere, had decided to weld the fucking battery case onto the phone. It took two grown men and several thin pieces of sturdy metal to pry the damned thing off.

The SIM was already installed. I plugged the battery in and went on-line to activate.

"Your phone is taking longer to activate than expected," the web page read after a five minute wait. "Please check again in 30 minutes."

Thirty minutes pass. I try again. Nothing.
An hour. Nothing.
Three hours. Nothing.

I went to bed hoping for better luck the next day.

"Congratulations!" the confirmation email greeted me Saturday morning. "Your phone has now been activated!"

All was good, except they'd activated a random number instead of my ported number. And the number itself, it seemed, belonged to a Texas gentleman. I had his full address and everything when I logged into the account manager.

I called AT&T back to play the ping pong between departments game for another fourty-five minutes. Unable to understand what was going on, they decided my SIM card must be bad. I'd have to report to a Cingular store to get it replaced.

Whatever. I was going to Costco that evening anyway. How long could it take to stop by the store?

It was like a scene from Grease inside the spacious Cingular store. I've never actually seen Grease, so I guess I should say it was like what I'd imagine a scene from Grease to be like. The movie was about shiny, sticky, disgusting grease, right? Because the Cingular employees were covered in it. I wanted to scrape it off their hair and make french fries.

Anyway, I present my problem and (after flashing his expensive company-provided phone at me several times) the customer service rep disappears into the back.

Thirty minutes later he re-appears, but not for long. I think he saw his shadow and ran back terrified.

"I think I already know the answer to this," he says upon his next return, "but how important is it that we port over that old number?"

Honestly I didn't care that much. It's not like people ever call me. But greasy things are entirely too fun to play with.

"Oh," I say. "Very."

The way every muscle in his face dropped a good inch made that two hour Cingular visit worth every second.

He calls up his corporate office, hoping to find someone who can help him. "I've worked here a long time," he explains, "and never once have I seen someone trying to port a number to a pre-paid cell phone."

I grin. I like being clever.

I make or receive around eight calls a month. At most. And it's rare that any of them last longer than two minutes. That's $4.00 / month. Even if you consider that your pre-paid minutes expire every 30 days and the lowest recharge amount is $15, that's still only $15 / month. It's a far cry from most cell phone plans.

After being put on hold for fifteen minutes he pleads with me again that I just give up the number. The system, he explains, simply wasn't set up for porting to pre-paid phones.

I refuse, of course.

At length he's able to find someone to port the number over. His smile returns! He'll finally be able to get rid of me!

His joy is short-lived, however. He found a problem. New Go Phones are supposed to be charged with $10 credit. Mine was, but this was applied to the old number--not the ported one.

"You want that $10, don't you?" he asks, reaching for his phone before I answer.

"Yup."

Another ten minutes pass and he proudly returns my re-credited phone.

I thank him.

He says, "Your welcome." What he means is, "Get the fuck out of here."

Whatever. At least I finally have my phone.

test

A Somali arrives in Minneapolis as a new immigrant to the United States.

He stops the first person he sees walking down the street and says,
"Thank you Mr. American for letting me in this country, giving me housing,
food stamps, free medical care, and free education!"

The passerby says, "You are mistaken, I am Mexican."

The man goes on and encounters another passerby. "Thank you for having
such a beautiful country here in America!"

The person says, "I not American, I Vietnamese."

The new arrival walks further, and the next person he sees he stops,
shakes his hand and says, "Thank you for the wonderful America!"

That person puts up his hand and says, "I am from Middle East, I am not
American!"

He finally sees a nice lady and asks, "Are you an American?"

She says, "No, I am from Russia!"

Puzzled, he asks her, "Where are all the Americans?"

The Russian lady checks her watch and says..."Probably at work."