DranoK
11-15-2007, 03:53 PM
Singularity, Part II: The Network Upgrade
Articles in this series
Singularity, Part I: Confessions of a Digital Pack Rat (http://blog.deadgod.net/2007/11/07/Singularity_Part_I_Confessions_of_a_Digital_Pack_R at)
Singularity, Part II: The Network Upgrade [Image 1 (http://liquidx.DeadGod.net/Nicks_Photos/Singularity/PDRM0011.JPG)] [Image 2 (http://liquidx.deadgod.net/Nicks_Photos/Singularity/PDRM0012.JPG)]
Singularity, Part III: Construction (http://linux.deadgod.net/2007/12/06/Singularity_Part_III_Construction)
Singularity, Part IV: Performance Testing (http://blog.deadgod.net/2008/01/25/Singularity_Part_IV__Performance_Testing)
Singularity, Part V: Conclusion
Home Gigabit Network
Gigabit networking has recently become an option for consumer budgets. This advance has been fueled by the proliferation of cheap, unmanaged switches by Netgear and Cisco.
Most of these were four-port or possibly eight-port creatures, however--most consumers didn't have a need for more than this. Businesses still preferred more expensive managed switches for their production network, and their office network often didn't need it. Especially if everyone is in love with wireless.
This is changing, slowly, as people become disillusioned with cubical wireless and trend back toward wired access. As storage continues to increase the uses for it explode. More and more people start to demand gigabit speeds.
Many corporations have already deployed gigabit managed switches for their office environments. Many small businesses, however, have not. Larger-capacity unmanaged gigabit switches cater to small businesses like these, as well as the enthusiast market.
The Gigabit Myth
People often run network tests and find they're not getting "true" gigabit speeds (128 MiB/s). Typically they blame the switch. To some extent they're correct--cheap, unmanaged switches rarely perform well in a single transfer vs. their pricier cousins. There are other reasons, however.
Even with a RaptorX SATAII drive (http://www.wdraptorx.com/en/specifications/), the absolute best-case transfer rate you're going to get (buffer -> disk) is 84 MB/s. I'm not sure if this is MB or MiB--regardless it's a far cry from 128MiB/s. And that's just your raw 10k disk speed. If your network ports are built into your motherboard you're going to have a slowdown any time your CPU is busy (like when it's buffering and writing disk data). PCI network cards are typically no better, unless it's a PCI-X or PCI-E with offloaded processing.
Where gigabit networks shine, however, is in multiple transfers. Sure, any single given device may not seem like it's taking advantage of gigabit--but when two or more devices are transferring data at the same time you will definitely notice a difference.
The New Network
We jumped to gigabit for the first time when we built Sol. We quickly ran out of ports, however, and have been hobbling along with a 16-port 10/100 Linksys trunked switch ever since.
With cheaper, higher-capacity gigabit switches now available we took this opportunity to scoop one up.
http://dranok.DeadGod.net/normalizedblog/Pics/Random/enus_left-lores_product_gs116.jpg
We also had a cabling rat's nest problem, as most people probably do. You never have the right length of ethernet cables so inevitably end up with 15ft monsters connecting devices which are two feet from each other. We decided we wanted to buy our own cabling kit and crimp new cables ourselves so we could better control the length. We found a good deal on a kit:
http://dranok.DeadGod.net/normalizedblog/Pics/Random/C184-28428-main.jpg
We especially enjoyed the Engrish instructions :)
http://liquidx.DeadGod.net/normalizedblog/Nicks_Photos/Singularity/cabletester.gif
Beyond Gigabit
Now that we had an excess of ports it was time to push Sol beyond gigabit speeds. Since we had two gigabit NICs available here we decided to aggregate them into a single, bonded device.
Network bonding is a method of combining two independent NICs into a single virtual device to provide both redundancy and higher throughput. Under Linux there are many different network bonding modes available, from simple round-robin load balancing to advanced load balancing. For our purposes we chose mode 6, or "Adaptive load balancing". The actual details of network bonding is beyond the scope of this document; more information can be found in your kernel distribution (/usr/src/linux/Documentation/networking/bonding.txt (http://www.id.ethz.ch/about/sections/kom/dkid/themas/link-aggregation/bonding))
Now our current fileserver is running at two gigabit while all our other nodes are at regular gigabit. We are now I/O bound on Sol, however this should speed up simultaneous access. As an added bonus, we can run new ethernet cables and swap ports without causing a network interruption.
Next: Singularity, Part III: Construction (http://linux.deadgod.net/2007/12/06/Singularity_Part_III_Construction)
Articles in this series
Singularity, Part I: Confessions of a Digital Pack Rat (http://blog.deadgod.net/2007/11/07/Singularity_Part_I_Confessions_of_a_Digital_Pack_R at)
Singularity, Part II: The Network Upgrade [Image 1 (http://liquidx.DeadGod.net/Nicks_Photos/Singularity/PDRM0011.JPG)] [Image 2 (http://liquidx.deadgod.net/Nicks_Photos/Singularity/PDRM0012.JPG)]
Singularity, Part III: Construction (http://linux.deadgod.net/2007/12/06/Singularity_Part_III_Construction)
Singularity, Part IV: Performance Testing (http://blog.deadgod.net/2008/01/25/Singularity_Part_IV__Performance_Testing)
Singularity, Part V: Conclusion
Home Gigabit Network
Gigabit networking has recently become an option for consumer budgets. This advance has been fueled by the proliferation of cheap, unmanaged switches by Netgear and Cisco.
Most of these were four-port or possibly eight-port creatures, however--most consumers didn't have a need for more than this. Businesses still preferred more expensive managed switches for their production network, and their office network often didn't need it. Especially if everyone is in love with wireless.
This is changing, slowly, as people become disillusioned with cubical wireless and trend back toward wired access. As storage continues to increase the uses for it explode. More and more people start to demand gigabit speeds.
Many corporations have already deployed gigabit managed switches for their office environments. Many small businesses, however, have not. Larger-capacity unmanaged gigabit switches cater to small businesses like these, as well as the enthusiast market.
The Gigabit Myth
People often run network tests and find they're not getting "true" gigabit speeds (128 MiB/s). Typically they blame the switch. To some extent they're correct--cheap, unmanaged switches rarely perform well in a single transfer vs. their pricier cousins. There are other reasons, however.
Even with a RaptorX SATAII drive (http://www.wdraptorx.com/en/specifications/), the absolute best-case transfer rate you're going to get (buffer -> disk) is 84 MB/s. I'm not sure if this is MB or MiB--regardless it's a far cry from 128MiB/s. And that's just your raw 10k disk speed. If your network ports are built into your motherboard you're going to have a slowdown any time your CPU is busy (like when it's buffering and writing disk data). PCI network cards are typically no better, unless it's a PCI-X or PCI-E with offloaded processing.
Where gigabit networks shine, however, is in multiple transfers. Sure, any single given device may not seem like it's taking advantage of gigabit--but when two or more devices are transferring data at the same time you will definitely notice a difference.
The New Network
We jumped to gigabit for the first time when we built Sol. We quickly ran out of ports, however, and have been hobbling along with a 16-port 10/100 Linksys trunked switch ever since.
With cheaper, higher-capacity gigabit switches now available we took this opportunity to scoop one up.
http://dranok.DeadGod.net/normalizedblog/Pics/Random/enus_left-lores_product_gs116.jpg
We also had a cabling rat's nest problem, as most people probably do. You never have the right length of ethernet cables so inevitably end up with 15ft monsters connecting devices which are two feet from each other. We decided we wanted to buy our own cabling kit and crimp new cables ourselves so we could better control the length. We found a good deal on a kit:
http://dranok.DeadGod.net/normalizedblog/Pics/Random/C184-28428-main.jpg
We especially enjoyed the Engrish instructions :)
http://liquidx.DeadGod.net/normalizedblog/Nicks_Photos/Singularity/cabletester.gif
Beyond Gigabit
Now that we had an excess of ports it was time to push Sol beyond gigabit speeds. Since we had two gigabit NICs available here we decided to aggregate them into a single, bonded device.
Network bonding is a method of combining two independent NICs into a single virtual device to provide both redundancy and higher throughput. Under Linux there are many different network bonding modes available, from simple round-robin load balancing to advanced load balancing. For our purposes we chose mode 6, or "Adaptive load balancing". The actual details of network bonding is beyond the scope of this document; more information can be found in your kernel distribution (/usr/src/linux/Documentation/networking/bonding.txt (http://www.id.ethz.ch/about/sections/kom/dkid/themas/link-aggregation/bonding))
Now our current fileserver is running at two gigabit while all our other nodes are at regular gigabit. We are now I/O bound on Sol, however this should speed up simultaneous access. As an added bonus, we can run new ethernet cables and swap ports without causing a network interruption.
Next: Singularity, Part III: Construction (http://linux.deadgod.net/2007/12/06/Singularity_Part_III_Construction)