Hey, everybody bitching about how close a call it is for 1TB, just make it 10TB. There. Problem solved. 100Mbps internet link now no longer even remotely close. Stop nitpicking, it's pointless. As the amount of data to move increases, moving physical storage around wins increasingly more, nitpicking won't change that.
The whole comparison is flawed/irrelevant anyway. Their point is valid irrespective of the airplane comparison in that having CDN servers closer indeed mean faster downloads, but I could have a perfect 100Mbps connection to the building across the street and try to download a 1TB file, it would still be insanely faster to travel to the other side of the road and lug the hard disk back.
Of course it's relevant... Any time you're considering moving a large chunk of data to a remote datacenter, it makes sense to do the math and determine whether it's faster/more cost-effective to move the data there digitally or physically. Why do you think AWS allows you to ship them your portable storage? http://aws.amazon.com/importexport/
This was actually utilized at some point in Usenet's history to transfer posts between the US and Australia, as I recall. Googling isn't turning up citable references, so I'll call this apocryphal until verified. I do find traffic stats showing ~4GB/day posts as of 1996 (http://en.wikipedia.org/wiki/Usenet#Technical_details).
Australia for a long time was a worst-case major economy networking situation with few, slow, and high-latency links (and still suffers from high costs due to limited links and telco monopolies).
For an understanding of how limited international data links were as recently as the early 1990s, one of the key public data links between the US and Europe at the time was a 9660 baud link according to a dated, but fascinating, compendium of networking at the time (lost somewhere in the Krell stacks, I'll dig a reference on request).
Jon Bentley's classic Programming Pearls discusses the problem of transferring graphical data images between Mountain View and Santa Cruz, California, in the 1970s. The high-bandwidth, cost-effective, sufficiently reliable solution? Carrier pigeons. Really.
> For an understanding of how limited international data links were as recently as the early 1990s, one of the key public data links between the US and Europe at the time was a 9660 baud link according to a dated, but fascinating, compendium of networking at the time
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. —Tanenbaum, Andrew S. (1996). Computer Networks. New Jersey: Prentice-Hall. p. 83. ISBN 0-13-349945-6.
A Volvo full of LTO-5 tapes still has a very respectable bandwidth. LTOs come in 20 tapes boxes, about 10 litres in volume; a Volvo break with backseat down fits about 1500 litres, so you should be able to fit 150 boxes of 20 tapes = 150 x 20 x 1.5 = 4.5 Petabytes.
There are actually special file transfer protocols for high-bandwidth high-latency situations where you want to transfer gigabytes/terabytes across the world.
The main example I've heard is special effects companies like Weta Digital in New Zealand wanting to deliver video to the US across a 200ms latency link.
There are a few open protocols in various stages of development or you can pay for a product like FileCatalyst http://www.filecatalyst.com/ ( some interesting case studies on their website ).
Yes, you're likely thinking about the Aspera range of file transfer software which is quite popular within the film industry.
It's actually quite elegant; a control connection over SSH plus a custom UDP protocol (FASP) for data. It does work as advertised and has some nice features that are handy in production, but licensing is very expensive ($20K for point-to-point at 1gb/s).
"There are a few open protocols in various stages of development"
Can you, or somebody else please name some? It always irks me a little when someone just mentions that "these projects" exist, and then fail to mention any of them.
We offer a custom protocol file transfer as a service (uses UDP instead of TCP), but you can do a home grown solution yourself very easily.
If you have a big file of compressed content (video), do the following. If you have a lot of small files, tar and gzip them, then do the following.
md5 your file
split your file into chunks (eg parts _a - _z, or _aa - _zz)
use ncftp or similar capable of parallel transfers
optimize between 8 and 30 parallel depending on latency to saturate link
concat the file at the other end
check the md5
You can handily saturate a link from EU to Australia or India to US this way.
That said, for Aussie customers sending us 1000s of movies, we FedEx or DHL pelican cases full of drives.
Aspera (http://asperasoft.com) is another one (but my god are they expensive.. the most basic setup possible will run you about $25K). I'm definitely interested if there are open alternatives.
I came here to say that my father works for a company called MTU Areo Engines as an FAA RS-DER. What that means is he can write and design repairs for the hot-sections of most engines. Why this is interesting, and related to this story is something that comes along with that.
All engine data that the pilot can see is available realtime on the ground at the repair stations.
What this means is that an engineer can call up a GE-CF6 in the second position on the starboard wing and see how it is performing. While that might not sound all that interesting, there are literally tens of thousands of measurements taken a bunch of times a second and transmitted to the ground for each engine.
For instance, if the flight was 10 hours (and every package was 100 Kb) they'd send about 35GB of data per engine. This is just the engine measurements. This does not include avionics, communication, position.
Most of the information is propriety for how they store, transmit, and calculate these things so I do not know exactly how this works. I have seen one of these workstations and it is really impressive to know the exact RPM of the number two engine compressor is spinning at in real-time of a flight crossing the Atlantic.
Also, you can two-way text-message with the pilot from this workstation to pass perceived information about the engine performance.
Back in the 90s there was a 'cloud' system for Rolls Royce engines.
Each engine logged data in flight to a 1Gb hard drive - the next time it landed somewhere with a RR service point the drive was taken out and plugged into a server and a fresh drive put in the engine.
The clever part was that most data would never be used - most of it was only needed if that engine developed a fault - so the database kept a central record of where each engine+date disk was in the world and only copied the data back to HQ if needed.
Now the plane has a sat data feed and the engine data goes live back to a monitoring station.
I wonder how suitcase-full-of-truecrypt-disks compares to pgp/tls transport of data over the internet in terms of probability of detection, interception and access.
I recall the US now has a relatively free hand in confiscating storage media for analysis/cloning during border crossings, and you might not get your copy back for a couple of months if they take a serious interest.
So, even though the aeroplane travel infrastructure itself is quite reliable, there are points of failure that would require another copy of the data to be created and shipped, with a much greater latency (It might take days to purchase more disks, transfer all the data onto them, find a willing courier, book their flight, etc).
A friend at a botanical institute sends and receives a 500GB hard drive weekly, exchanging data across the Atlantic. Back of beermat calculations at the time demonstrated huge efficiency compared to similar cost broadband solutions!
Eyup, not sure why this blindingly obvious truth has to be rediscovered once in a while.
This is also how astronomers work, radiotelescopes produce terabytes of data per day (the LCH produces ~15PB/year) so sending that over the wire makes absolutely no sense. Instead, they ship external disks for smaller datasets[0] and for bigger ones some go as far as shipping complete NFS/CIFS servers[1].
This also makes the assumption that you'd have zero time copying the data once you arrived in London. More realistically, you'd probably make a copy. In this case, you need to also take into account the amount of time copying the files at the other end. Depending on your external drive's interface, this is a non-trivial amount of time.
Well if we're speaking more realistically, you'd use this for shipping 100tb worth of disks (which would fit in your luggage in both size and weight), and it would blow away just about any internet link you could lay your hands on with a remotely similar budget.
That is a great point though. A 4TB drive, hooked up to your internal SATA port, will do about 130 megabytes per second, which is around 9 hours for a full copy.
Drive speeds increase linearly with track density, but drive capacity increases quadratically, so this is only going to get worse. Hard drives are the new tape.
For a point of reference -- LTO5 pushes data at up to 140 megabytes per second.
(But Tape has the same problem -- Capacity increases faster than speed)
If you have the data on 32GB micro SD cards (at about three to the gram), you're at 96TB/kg, or 2.88 exabytes/747. On a ten-hour trip, that's 80TB/s. It wouldn't be at all practical to ship 'em loose, and finding and using the data you want from the "stream" would be a real SOB, but that's a lot of raw throughput with current, non-exotic tech.
Of course it's only the end-to-end throughput that counts. It might take quite a while to load the data onto all those micro-SDHC cards and then extract it again at the destination, and you have to add time that to the ten-hour flight time. Not to mention the health-care bill for driving your sysadmin insane.
My point, really, was that even with inefficient and inappropriate packaging, the available raw transfer rate is already huge. It's just a matter of packaging what already exists into "data cargo containers" sized and priced appropriately.
http://wdc.com/en/products/products.aspx?id=120#tab3 says 3TB weighs 0,74 kg. So that would make 40.540 drives -> 121,620TB in 10 hours. But I'm not sure they would all fit due to their volume relative to airplane's.
In either case, it's an interesting exercise but missing some key calculations...
When I transfer data from machine to machine, the data is immediately "usable" on the receiving machine.
When transferring data by physical means, the data needs to be brought to the final point of consumption, and possibly loaded into the host machine if it is not in some form that could be immediately connected/mounted.
So, the real comparison would be transferring data from the host machine to some form of removable media, taking that media to the airplane (early enough to meet all the pre-clearance stuff), waiting for the plane to takeoff, fly, land and then handoff the physical media at the other end.
In many cases, I'm sure the plane is still faster, but if we're talking about its theoretical bandwidth we need to look at the entire "trip" of the packets/data, not just one segment of it.
Unless you take the rackspace crates off the 747 and hook them up at the destination end. But compare that to ordering blank disks, having them delivered, configuring them and then copying network data onto them.
I guess that you were right and you do not need to change anything. 220 Mbps is the throughput and not the bandwidth.
Assume a scenario wherein you change the destination to Mumbai. Your throughput will get reduced to 110 Mbps (assuming a 20 hour flight), while the bandwidth will stay the same as the networking medium (airline) has not changed, right?
Couple of other points -
* Two media with same bandwidth can have different throughput depending on the distance (i.e latency) or packet loss or other reasons. Consider two 100 Mbps broadband lines from SFO to London. One goes via the east coast, while other via Asia. Both will have different throughput; difference being roughly proportional to the difference in distances.
* 220 Mbps would imply that a maximum of 220 Mb data can pass through the medium, which is not the case, since we are loading TBs of data on the plane.
* The window size in case of an airplane would be the size of the data loaded on it i.e the size of the movie collection.
Out of curiosity, what caused the error in the article?
I was wondering if you derived the number via a similar calculation as above (a full 747), it's a direct reference to the size of the hard drive it's carrying, or just a simple typo?
It was an editing error. I originally had a whole paragraph imagining a continuous line of 747s containing 1TB disks and was going to be making an analogy about the connection across the Internet, but it was too complicated. In editing that out I didn't go back to the right number.
I'm assuming 747 option is using carry on luggage otherwise at one stage the underlying infrastructure changes to packet transfer by baggage handler which, in my experience, can be an incredibly high latency protocol with a possibility at times of zero error correction.
28 crashes a day is a lot. But author assumes that storage magically appear inside 747 during takeoff add disappears during landing.
But if we will include this strange institution called customs then 28 item loss per day won't be unrealistic at all.
Items detained indefinitely in the customs, items detained until bribe is paid, items lost, items stolen during load/unload, items broken (hits, pressure changes, magnetic gateways and handheld scanners etc.). Lots of problems in general, all are multiplied if your country customs and airport personnel are incompetent.
However, there is one thing that the delay (in the cited article) does not take into account is that the flights do not take-off all the time, so if you have data ready to be sent and no one to take it, then there is waiting delay before it can be sent. so if there is only one flight in a day then some data will have to wait between 24+10h to 0+10h before it can arrive. Since storage cost is very low, one could ship a lot of data but this data cannot be delay sensitive else.
on the other hand, using TCP one can prioritize data and send more important data before the rest. Of course there may be intrinsic problems with TCP receiver window and one of the proposal is to increase the initial window but the research community and the IETF is a bit split on the real benefits of it. The fear is that it may cause a congestion collapse. Moreover, the current buffer-bloat in the routers and the increase in the window size may only aid in causing a congestion collapse (http://tools.ietf.org/html/draft-gettys-iw10-considered-harm...).
While it makes for a good blog, this idea ignore cost ($1200? to transfer 1tb data) and it ignore that you still need to get the 1Tb _off_ the drive once you get to London.
Typically drop shipping something on an airline is substantially cheaper than a full fledged seat fare. You go to the counter, say you want to ship something, they toss it in the belly and your friend picks it up at the counter at the destination. We used to do this all the time with equip in Alaska.
I enjoyed the premise behind this post, and perhaps this is pedantic or I am simply confused, but why does the it conclude with, "The speed of light really is a limiting factor in downloading!" and talk about what Cloufare does to "fight the speed of light" after smartly explaining network latency at the layers above the physical one?
I guess fighting the speed of light sounds cooler, but that's not really the problem they are solving or speaking too, is it?
All this makes me think of is "In order to transmit the same amount of information on paper, they would have to arrange for a 747 cargo freighter packed with telephone books and encyclopedias to power-dive into their unit every couple of minutes, forever."
OK... but the nice thing about computers and networks is that you don't _necessarily_ have to transfer 1TB globs of data to from SF to London to use it or compute upon it. There are computers in SF too. And of course, if you're talking about media files you can't watch more than one at a time and anyway you don't need "random access" to the media file (you can stream them).
1. TCP wasn't designed like that. It uses a rather simple scheme of sending back a byte count telling the other side up to what byte number it has received. So there's no provision for telling the other side that a bit in the middle is missing.
2. We don't send 1000s of packets at once because doing so would cause congestion on the Internet. So there's a whole subsystem in TCP that tries to determine the capacity of the link between two points and avoid creating congestion.
Note that SACK is enabled by default on all modern TCP implementations, so #1 really isn't an issue.
And the 1000's of packets thing isn't quite right either. With windows scaling (another option pervasively enabled) and a fat pipe (it does take some time for the algorithm to ramp up that far), you can have about 1GiB in flight on the wire at any time, that's hundreds of thousands of packets.
Because your LAN connection is probably 1000Mbps and your internet connection is likely <5Mpbs. Or the receiving sides internet connection is 2Mbps. Your computer only sees 1000Mbps and would dump a 1tb file out at line rate. Then it would ask 'what pieces are missing?' And have to send 99% of the file again. Even worse if multiple people are trying to send to the same recipient.
So instead we ask the receiving side how big its buffer is and only send that much before awaiting a response (in many cases more than 10 at a time). If there is network congestion that response will take longer automatically slowing the transmission rate. The problem is that TCP can't tell the difference between congestion latency and distance latency.
This calculation works if you can directly plug in the disk and start using the data that way without copying. If, when the disk arrives, you still have to plug it in and copy it at 50MB/s via an IDE cable then that has to be factored into the calculations.
The analogy would have worked better if you used a Fedex jet instead of a BA one. That said, I don't think the latency of transferring large data is the best selling point for Cloudflare, especially since the acks are asynchronous and good load balancers perform quite well over long latency links.
A better selling point for distributed delivery is the round trip when you deal with multiple transactions, for example requesting a web page and then the included objects, or submitting data and waiting for a response.
Given that the OP doesn't factor in delays due to transit to and from the airport, clearing immigration and customs, and other delays, a more apt comparison would be the relative bandwidth of transferring 1 TB from one part of town to another. At that point, you're only facing traffic. If it takes 1 hour from one part of the a town to another part of the same town (4 if you are in LA), then the relative bandwidth is much more remarkable.
See, I always told my friends it was faster to burn to CD and then toss the disc their way. I mean, the last time I had to make a null modem cable it took me hours just to find the right tools!
With friends we do the same when we meet to LAN play. We copy the games into a USB stick and pass it around. It's waaaaay quicker than downloading from steam, and quicker than copying through the local net.
If you know the right folder you just copy it there, buy the game if you didn't already have it, and just wait a little for it to check the integrity of the install.
This article has me almost wishing to start a data courier service. Flying around the globe transporting data.
Quite common on translatlantic routes.
Cruise on a 747-400 is about mach0.85 (925km/h) and you commonly have a jet stream of 150km/h so a ground speed of close to 1100km/h is possible - above the speed of sound at 30,000ft
I learned this in my Aero Engineering courses in college. My professor mentioned that occasionally if the light hits the wing at the right angle, you can actually see the shockwaves.
But for all my years of flying since, I've yet to see it... :(
Under the right conditions you can expect to see some puffs of condensation due to the adiabatic cooling of the air on the edges of the shockwave. But then only under the exact right circumstances (moist air, sun from behing the viewpoint, relative high pressure).
3TB drives weigh ~1 lb = 180,530kg/1lb * 3Tbytes / 3.5 hours in seconds ~= 235TBytes/s which is rather close. Especially when you consider how long it takes to load those container ships and how limited the destinations are.
PS: As to pirates your 84 million drives at 150$ a piece would be worth ~12.6 billion dollars which might actually be a tempting target even if you can only get 10% of their actual value in ransom.
Edit: Even this is small next to MicroSD cards. ~64GB/gram = 29Tbytes/lb = 2,270Tbytes/second across the Atlantic for ~12.64 billion.
= 1940.10 miles at 600MPH your at 3 hours 14 minutes not that there are any direct flights between those airports, but if your spending 12 billion on micro SD cards, I think you can charter an airplane and spend extra fuel going a little closer to top speed vs cruse.
This doesn't take into account UK customs. I recently send a nicely packaged 1Tb drive to my place in tb uk by express mail from San Francisco. The drive arrived in customs two days later, sat there for a week, then they mailed a postcard saying the import duty was 550 GBP. Now we are up to about the third week... Obviously I didn't pay that so the drive went back, taking over two months. Using scp would have been far quicker...
So next time you're considering physical transport include the time spent in the destination country's customs, with their highly incompetent and inefficient crews.
First off, do you really think it's more practical to ship data per airplane than a simple cable that's already connecting you? Not that bandwidth is completely for free, but most of the time you're paying a flat fee and transferring this 1TB won't cost you anything extra. I bet that Boeing is quite a bit more expensive.
Secondly, at some point the article assumes the 'transfer rate' of the Boeing to be 222Mbps. While this might be true for this practical example--let's ignore that you actually need to get the data on the plane in the first place--I bet that you could load it up from bottom to top and reach practically a speed of many gigabits ifnot terabits per second.
Thirdly, the Boeing flies with physical stuff. If the plane crashes, the data is lost. You would need backups, which take time. Over the internet it's not even possible to transfer the original medium; you'd need to first transfer and then delete it.
Fourthly, while it might be true that a packet gets corrupted 28 times during this 1TB transfer (I myself do not have statistics on this, but let's assume), on the internet you can retransmit a packet. Try retransmitting that one sector on the harddrive in under a second.
Fifthly, the article goes on about how TCP needs to wait for an acknowledgement (or 'ack' in short). I've made this mistake before when choosing for UDP instead of TCP because "it would not have to wait for acks and thus make my game much faster". It's wrong because there is a so-called congestion window. I do not know the exact figures, but it's like this: The server sends packet one. Packet one is in transit. The server sends packet two. Packet one and two are in transit. The server sends packet three. Packet one arrives. Packet two and three are in transit. The server sends packet four; the client sends ack 1. Packet 2, 3, 4 and ack 1 are in transit. Etc. The thing is that the server can send up to X packets ahead, without receiving an ack.
Sixthly, the article finally concludes that this is why you should choose for Cloudflare--so it was an advertisement after all. That explains why it's told in this story-like way and why it contains so many technical errors.
1. Yes, physical shipping is more practical than using the Internet for large transfers and this is done frequently. For example, Amazon Web Services offers the possibility to send them disks: http://aws.amazon.com/importexport/ Shipping of disks is also common in the movie industry. For example, a lot of digital movie distribution to cinemas relies on shipping of hard drives.
2. I used a statistic of 0.1% packet loss on the Internet and 28,000 flights per day to arrive at 28 flights lost per day.
3. There are two interacting things happening in TCP: the congestion window (which limits the number of outstanding _packets_) and the receive window which limits the number of outstanding bytes. This blog post does not address the congestion window at all and I assume that the receive window can always be filled. Thus is _overestimates_ what might be possible on a real link.
4. For very large transfers there are improvements that can be made using UDP and there are protocols that do just that. See, for example, TixelTec.
> Shipping of disks is also common in the movie industry.
As well as sciences based on massive amounts of data (astrophysics and particle physics for instance, where experimental systems can produce terabytes per day), financial data analytics, or petroleum seismic surveys.
Amazon allows you to send them hard drives if you have lots of data, so yes, it's more practical.
You should calculate how large a window you need to fill a 100mbps pipe between SF and London. Then calculate how likely it is to remain there before you drop a packet, and how long it takes to detect a dropped packet with typical timeouts, and so on.
