Why Usenet File Sharing Works So Well

The amount of data that must be stored to provide Usenet's file sharing service is enormous, and most Usenet providers never throw anything away. In other words, they save — permanently — all the data that is uploaded to Usenet. This concept is called RETENTION and is measured in days.

For example, if a provider offers 1,000 days of retention, it means you can access anything that has been posted to Usenet in the last 1,000 days. Retention of text files is considerably longer, because text files require much less disk storage than do binary files.

As we discussed in the previous section, most of the data within Usenet consists of binary files that are uploaded so as to be available for sharing. How much data is this? As I write this, the amount of data uploaded to Usenet is in the order of 9 terabytes a day. This means that for a Usenet provider to retain everything that is posted to Usenet, they must store over 9 trillion bytes of new data every day! Moreover, the total amount of data — year's worth — must be available quickly, at all times, to a vast number of simultaneous users.

To provide such service, Commercial Usenet providers build their systems around SERVER FARMS: complex data-storage systems consisting of a great many special-purpose computers designed to act as file servers. Server farms are generally very large, very fast, and very reliable. Usenet server farms are designed to support high-speed uploading and downloading for a large number of users at the same time. Indeed, the largest Usenet providers actually maintain several server farms, each of which consists of many computers and disk drives working together as a single, immense entity.

In order to appreciate just how remarkable the modern Usenet system is, let's compare it to BitTorrent, another very popular file-sharing system.

Like other Internet-based systems, BitTorrent is a client-server system. There is, however, a big difference between Usenet and BitTorrent. When you use Usenet, your client program (that is, your newsreader) connects to a large, centralized news server, maintained by an Usenet provider. BitTorrent, however, is designed to not use a central server. Instead, BitTorrent users share data by connecting directly to other users's computers.

In other words, where Usenet users upload and download files from a central server, BitTorrent users upload and download files from one another. Such a system is called a PEER-TO-PEER network, often abbreviated as P2P. Each individual computer (including yours) is referred to as a PEER.

The advantage of BitTorrent is that it does not require anyone to maintain an expensive server farm to store massive amounts of data. Thus, BitTorrent access is free and — because you don't have to sign up for anything — relatively anonymous. There is, however, an important disadvantage: because there is no central server, BitTorrent is a less reliable and a lot slower than Usenet.

As an example, I made several tests using both BitTorrent and Usenet. First, I downloaded a very large software file (the latest version of Ubuntu Linux). Next, I downloaded a very large video (a movie). Finally, downloaded a collection of music (an album). You can see the results in the table below.

Downloading Tests: Comparing Usenet and BitTorrent

	Download Times

Type of File	Size	Usenet	BitTorrent	Speed Ratio
Software (Linux)	694 MB	8.3 min	19.5 min	2.3
Video (Movie)	667 MB	12.3 min	173.4 min	14.1
Music (Album)	56 MB	1.2 min	38.1 min	31.8

The first column shows the type of file that was downloaded. The second column shows the size of the file. The next two columns show how long it took to download the file using Usenet and BitTorrent respectively. The final column shows how much faster Usenet was than BitTorrent. For example, when downloading a 667 megabyte movie, Usenet was 14.1 times faster that BitTorrent.

Downloading speeds vary a lot depending on conditions, so I don't want to spend much time analyzing the numbers. The main point I want you to realize is that downloading with Usenet is significantly faster than with BitTorrent. (We'll discuss why in a moment.)

During the first test, it happened that the BitTorrent conditions were as good as they ever get (I'll leave out the details). Nevertheless, Usenet was still more than twice as fast. During the other two tests, Usenet was 14.1 and 31.8 times faster than BitTorrent. In my experience, these two results are far more typical and represent the type of performance you will probably see.

Interesting digression: What would happen if you used BitTorrent to download something huge? As an experiment, I used BitTorrent to download a very large library of video files, 21.1 gigabytes in all. The total download time was 3.2 days (77 hours).

My overall point is not so much that BitTorrent is slow. In fact, BitTorrent is an amazing file sharing system (as long as you are patient). What I want you appreciate is that — compared to other file sharing systems — Usenet is very fast. To see why this is so, we can ask the question: Why is there such a difference between downloading with Usenet and downloading with a network like BitTorrent?

When it comes to file sharing, Usenet has five important strengths. They are:

• Dedicated servers
• High-speed connections
• Dedicated downloading
• Centralization
• Availability of the entire file

Let's take a few moments to discuss the details of each of these strengths, one at a time.

Dedicated Servers

Usenet:
You connect to a fast, specialized, dedicated server that is designed and optimized for downloading large files. Such servers are always available, 24 hours a day.

BitTorrent:
You connect to a P2P (peer-to-peer) network in which all the computers involved are ordinary computers, such as the one you are using right now. You are, literally, downloading data from other people's personal computers. This is an important limitation in that you can connect only to those computers that are available at the time.

High-speed Connections

Usenet:
Dedicated Usenet servers offer extremely fast connections. To maximize the speed, you newsreader (client program) is allowed to have multiple connections with the server at the same time. Typically, you can have up to 20 connections. Your newsreader will take advantage of the multiple, high-speed connections automatically. Because the server is so fast, your downloading time depends on the size of the files and the speed of your Internet connection.

BitTorrent:
You connect to other people's personal computers, which means you are sharing their bandwidth. Thus, downloading is much slower, because it depends on the connection speeds to all the computers from which you are downloading.

Dedicated Downloading

Usenet:
When you are downloading a file, your newsreader (client program) concentrates only on downloading.

BitTorrent:
    BitTorrent does not use commercial servers. Instead, all BitTorrent programs are designed to act as both a client and a server, which means that your BitTorrent program downloads and uploads at the same time. Here is how it works.
    Whenever, you want to download a file, BitTorrent breaks the file into pieces. Each of these pieces is then downloaded separately from whichever computer on the network happens to have that piece and is available for a connection.
    Once you start to accumulate pieces of the file, your computer automatically becomes a source for other computers that need those pieces. In technical terms, your personal BitTorrent program acts as both a client (requesting the pieces you need) and a server (offering pieces you already have). This is a brilliant design which, in fact, is what makes BitTorrent possible in the first place: uploading while you are downloading ensures that everyone who benefits also helps support the system. This, of course, is only fair.
    From a selfish point of view, however, it does demand extra resources from your computer, which can slow down your downloading.

Centralization

Usenet:
All your data comes from a single server.

BitTorrent:
When you download with BitTorrent, the data comes from multiple computers, which can be anywhere in the world. As an example, during the second BitTorrent test I mentioned above (downloading the movie) I could see that, at a particular time, my BitTorrent program was connected to 33 peers (other computers) around the world. The breakdown was as follows:

   Australia: 1
   Brazil: 3
   Canada: 1
   Czech Republic: 1
   Italy: 1
   Mexico: 1
   Romania 1
   South Africa 1
   Sweden: 1
   United States: 19
   unknown: 3

TOTAL: 33 peers

Of these 33 peers, 14 were downloading to me, 6 were uploading from me, and 13 were both downloading and uploading at the same time (as was my computer).
However, my computer was not connected to these same 33 peers the entire time I was downloading. BitTorrent is a volunteer network, and a peer will disappear suddenly if the remote computer is turned off or if the BitTorrent program on that computer is stopped. This, of course, slows down the downloading process. On the other hand, new peers can appear just as suddenly, when a new computer that happens to have the file you want joins the network.

Availability of the Entire File

Usenet:
Both Usenet and BitTorrent break large files into pieces. (We'll talk about the Usenet details later in the tutorial.) As a general rule, if you find a file you want to download from Usenet, all the pieces of the file will be available, because everything you need comes from a single Usenet server.

BitTorrent:
Much of the time, all the pieces you need will be available. However, this is not always the case. From time to time, you will find that the peers (other computers) to which you are connected — in the aggregate — do not have all the pieces needed to re-create the original file.
For example, let's say a file is broken into 100 pieces. It may be that 5 of those pieces are simply not available anywhere. Your BitTorrent program will download the 95 pieces that are available, but it has no way to get the last 5 pieces. In such cases, you will have to wait until a peer that has what you need comes online, which can take a long time. Sometimes it never happens, which means your download will never complete.

To summarize: here are the reasons why Usenet file sharing works so well:

• Very fast
• Multiple connections
• Vast storage facilities
• Complete files (most of the time)
• Binary files are kept indefinitely
• Professionally maintained servers
• Organized into categories (newsgroups)
• Always available
• Secure (SSL) connections are available

So does that mean you should only use Usenet and not BitTorrent? Actually, I think you should know how to use both.

— hint —

Usenet has a lot of advantages over BitTorrent, particularly speed. However, you will often find that a file for which you are looking is not available on Usenet, but can be found using BitTorrent.

For this reason, if you are serious about file sharing, I recommend that you learn how to use both Usenet and BitTorrent.

(Remember, BitTorrent is free.)

At this point, we have covered the basic ideas regarding Usenet, and we are now ready to get into the details of file sharing. In the following sections, I will explain how Usenet file sharing works and what you have to do to upload and download files. Before we get started, however, I'd like to take a few moments to discuss an awkward, but important question.

Jump to top of page