Usenet News

Usenet News is probably the largest and most popular resourse available via an internet connection. Almost every internet user will want to access one or more usenet news groups.
This means that an ISP will most likely have to provide this resource to their users either by having their own news server or making some arrangement for their users to access someone else's new server.
Unfortunately dealing with news is neither a simple or inexpensive task and there are some issues that need to be considered:

Providing news to users

Users of your ISP will probably want to read certain usenet newsgroups. They generally have the expectation that access to news will be available and that it will not cost anything - or at least it is bundled into the costs of being a user on your system.
There are two options here for an ISP, either decide that you are not going to provide a local news server, inform your users, and let them try to access a news server elsewhere on the internet, or set up a local new server, arrange to get a newfeed from your provider or elsewhere and make news available from that server to your users.
So why would an ISP consider one or other option?

Using some other provider's news server seems a good idea, cheap and easy. However, the problem is finding a provider who will make their server available to your users. The cost and maintenance of a news server means that there are fewer and fewer available for public access and very few providers who will even allow reading access to the customers of downfeed sites. You can try asking your service provider if they have a news server that your users can access but expect the answer to be "No". Your provider will probably want to send you a newsfeed to your news server.
The only problem with using that other person's news server is that your users will have to use your link bandwidth just to read news or when they arrange a newsfeed then there could be a fair proportion of your main link's bandwidth used to transfer news directly to users. There is also a good chance that a lot of these newsfeeds will cover the same newsgroups hence wasting your precious bandwith moving multiple copies of the same news items.

Setting up and running a new server is not an easy or low cost task.
News servers require large disk space although the actual disk space will depend on the number of newsgroups stored on the news server and the time (usualy in days) news items are retained. This in itself is not often a problem as the falling price and increasing capacity of disk drives trend continues.
The volume of news on a server that attempts to take anything near a full news feed, plus the number of users trying to access news at any time, plus the number of news downfeeds being provided, will all push the server's disk interface to maximum capacity. This often means that the bottleneck in a news server will be the speed of it's disk interface therefore news servers tend to use the latest and fasted disk interfaces, often just to keep up with providing a reasonable news performance.

Where to get news?

The first place to look for a newsfeed should be your internet provider. Normally this should be your source of a newsfeed as your provider will want to minimise the volumes of usenet news using their link to the internet.
If they do not have a news server (unlikely!) then they should be able to point you to a news server that can provide you with a newsfeed.

All major providers in New Zealand maintain news servers. The best known news service is provided by Netlink who took over the news service provided by Victoria University (in associated with several other universities).
This was the first major news feed brought into New Zealand from 1989 and has been the major source of newsfeeds since that time. Many ISPs still subscribe to this service although in recent years alternatives for news feeds have become available.
The Netlink service is not free and Netlink have a monthly fee based on the size of the newsfeed they provide to a customer. The Netlink news service has about 7000 newsgroups available although in the recent past they have decided not to provide many binaries and sex related newsgroups. There is often no public discussion about these matters, or discussion with Netlink news customers, and final decisions over what newsgroups will be provided is left to the news administrators at Netlink. (see http://www.netlink.co.nz) The current pricing from Netlink is: Large newsfeed NZ $300 per month Medium newsfeed NZ $150 per month Small newsfeed NZ $ 50 per month

Largely because of Netlink news censorship there are now several other sources of newsfeeds coming into New Zealand.
One group of ISPs in Auckland now take a newsfeed from MCI, sharing the actual importing of newsgroups between three or four data links and local news servers. These ISPs then share the news between them using local data links. This newsfeed adds some extra 3000 newsgroups to those imported by the Netlink consortium.

There are other commercial news services available. Examples include Clarinet and APC newsfeeds.
Clarinet is a newswire service reporting on latest news items under various newsgroup headings. This service can cost substantial US dollars where the charges are based on the number of Clarinet newsgroups taken and the number of end users who are able to access the news. (see http://www.clarinet.com)
The Association of Progressive Communications (APC) is an umbrella organistaion that brings together many smaller news services that specialise in providing newsgroups that cover many specific areas such as ecology, labour issues, women's rights, housing, etc.. To subscribe to these newsgroups you need to become a member of the APC or be associated with an ISP that is a member. APC newsgroups are (were?) imported into New Zealand by the PlaNet New Zealand groups of ISPs and made available to PlaNet members. It may be possible to purchase these newsgroups from PlaNet. (see http://www.chch.planet.org.nz).

Various other ISPs throughout New Zealand have connections to Australia, Japan, Korea and Canada/USA for newsfeeds. Foreign language (non-English) news is also available from several sources.
Your service provider should be able to provide you with information on newsfeed costs and availability.

Bandwidth needed for a newsfeed

This will depend on the number of newsgroups you decide to subscribe to and the data volumes in those newsgroups. The impact on your main internet link will also depend on the bandwidth of that link.
Many ISPs start with a link of some type that has 48K or 64K bandwidth so if a news server is to be provided for users then planning the workable news volumes, newsgroups to subscribe to, and bandwidth requirements should be considered.

A near full news feed, about 7000 newsgroups in New Zealand, will amount to between 600 and 1200 megabytes of news items per day. Weekends tend to generate the higher daily news volumes with midweek days generating the lowest. These sort of volumes will have a substantial impact on a low speed internet connection, for example a 48K/bit MDDS link will have about 15Kbits/sec to 25Kbits/sec of bandwidth consumed by a full(ish) newsfeed, this meaning that from 30% to 50% of total link bandwidth could be used. To make matters worse the higher volume news days at weekends also tend to be the busiest days for an ISP so when maximum bandwidth is need for your customers the load imposed by a newsfeed is at its maximum.

Other news delivery mechanisms may be available to reduce the impact of a news feed at busy times. Your news provider may be prepared to only send news to your new server at offpeak times, usually during the early hours of the morning. Such a scheme may not be practical if your news server and link bandwidth has difficulty keeping up with your normal newsfeed, although for small to medium newsfeeds (hundreds to two thousand newsgroups) this should not be a problem.

News Server hardware

There is no specific computer hardware that makes a usenet news server.
The requirements of a news server computer will vary depending on the volume of news to be maintained, the size of the incoming newsfeed(s), the number of outgoing newsfeeds and their sizes and the number of online news reading sessions there could be.
A small news server will operate quite adeqately on a PC system. Even sharing a PC that is used for other tasks - eg dns/mail server, terminal server, web host etc.

News Management Software

The good news here is that good news management software is free. Several packages are available for just about every operating system and instructions for installing them are often straighforward although sometimes requiring a fair amount of configuration.
One of the most popular packages is INN, this is available from several FTP sites, the main site being News delivery Methods

News Readers

Fortunatly just about all of the popular web browsers now have a news reading capability. Web users need only set up their news configuration to point to a news server, get permission to use the server, and thats it.

News downloading programs for users

Below are some details from recent news postings about several packages.

NNTPRelay [ NNTPRelay is a usenet news router, something like Cyclone or Diablo. It runs on Windows NT. ]

We have released NNTPRelay 1.1 beta 1 of NNTPRelay. You can retrieve it from ftp://ftp.maxwell.syr.edu/nntprelay/

Primary contents of this release:

* new precommit caching code
* improved stability
* minor bug fixes
* a "console" for online configuration via telnet (not persistent yet)

All of the configuration for NNTPRelay is done via the TCL scripting language. The current version uses TCL to parse configuration files and setup sites, and the telnet console is implemented in TCL. We are planning to release new code in the coming weeks to allow persistent online configuration.

More information on NNTPRelay is available at http://nntprelay.maxwell.syr.edu. More docs will be online there shortly.

SLURP

I use slurp (I got my copy from ftp.demon.co.uk/pub/news/slurp) to collect news articles from a remote server and feed them to my INN server. To quote the slurp man page:

Slurp is an advanced passive NNTP client for UNIX. It will connect to a remote NNTP server and retrieve articles in a specified set of Usenet newsgroups that have arrived after a particular time (typically the last time it was invoked) for processing by your local news system.

NewsX

NewsX is an alternative to the "slurp" and "suck" already mentioned. It came to be since the author found the other alternatives unsuitable for various reasons.

NewsX was meant to be efficient, "polite" towards the news source, as well as easy to set up. YMMV of course.

There is a home page for NewsX at:

http://home.sn.no/home/egilk/newsx.html

The source code can be found at sunsite.unc.edu and any of its mirrors.
Finally, beware that news "pulling" only is a suitable approach if you intend carrying only a limited number of newsgroups on your local spool. Since you have a dial-up connection, I assume that is your intention anyway.

Newsgroups

All of the large world distribution newsgroups are available in New Zealand and these really form the backbone of a news feed. These are imported by Netlink and distributed to subscribers who can connect to one of their news servers.

The major news groups hierarchies are:
alt - alternative, the largest hierarchy that covers just about any subject, high newsgroup turnaround and some of the highest volume newsgroups are here especially the alt.binaries newsgroups and the alt.sex newsgroups.
alt can account for almost 50% of the newsfeed available in New Zealand.
rec - recreation, covering all types of hobbies, pets and sports areas.
sci - science, a limited number of science related newsgroups.
misc - miscellaneous, a few general sub groups the largest being 'forsale'.
news - usenet news software, management and announcements newsgroups including FAQs for nearly all other newsgroups.
soc - social newsgroups, the largest sub group is culture with almost every country and ethnic group having its own newsgroup.
comp - computing, almost all areas of computing from hardware to operating systems to programming languages are included, second largest after alt.
talk - covers specific chat areas which tend to be hobby or specialist activities.
gnu - the gnu newgroups exist to support gnu software products and product development. biz - business related newsgroups, often there are newsgroups for specific business products especially computer software.
list-serv - newsgroups that are mail list postings under various broad topic areas.

Country specific newsgroups.
Apart from New Zealand newsgroups the other country specific news hierarchy that is fully available in New Zealand is the Australian newsgroups under under the aus hierarchy.
There are about 50 newgroups under aus and a fair number of posting originate from NZ news sites.

New Zealand newsgroups

This the nz newsgroup hierarchy, presently consisting of sixteen or so separate newsgroups.
These are:
nz.archives - very low volume, (2 to 5 messages a week) postings mainly about New Zealand ftp sites and newly available files
nz.arts - low volumes, (5 or less messages a day) arts related topics
nz.biz.misc - moderated, quite low volumes, miscellaneous business topics/announcements newsgroup, see nz.biz.misc FAQ
nz.comp - medium to high numbers of postings (20 to 30 a day), computing related topics but does digress quite often.
nz.general - quite high volume, the 'catch all' newsgroup where anything and everything is posted
nz.net.admin - very low volumes, occasional messages limited to sys admins and those involved in NZ internet routing/administration
nz.net.announce - very low volumes, moderated.
nz.org.isocnz - few postings, newsgroup associated with the Internet Society.
nz.org.net-society - almost redundant newsgroups originally set up for the now almost defunct network society.
nz.politics - medium volumes (10 to 20 messages per day), New Zealand politics.
nz.rec - low to medium volumes (5 to 10 messages per day), recreation.
nz.soc - low volumes, anything on social issues.
nz.soc.queer - newsgroup for the queer community, low to medium volumes.
nz.soc.green - green/ecology issues, most messages from Alan Marston of Auckland PlaNet, low volumes.
nz.test - test newsgroup, low to medium usage, messages posted here get robot replies from about six news servers around New Zealand.
nz.wanted - really wanted and buy/sell newsgroup, medium to high volumes.

nz.* Usenet Hierarchy FAQ

Local or Private newsgroups

Many ISPs create their own set of newsgroups with distribution restricted to their own users or downfeed sites. If you have a news system already set up and operating then adding some local newsgroups is fairly easy and will provide a local content to news.
The good thing about local newsgroups (for the ISP) is that the normal procedures for creating a newsgroup do not apply since the newsgroup distribution will be limited to the ISP. Such local newsgroups often include:
A general chat group (general) which can cover any topic users want to discuss.
An announcements group which the ISP could use to provide service updates and users can also use to post their own items of interest. If the ISP announcements are regular or considered worthy of a separate newsgroup (perhaps read only) then often a service newsgroup is created.
A comp group or perhaps new-user newsgroups where the messages tend to be mainly computing related.
Maybe a test group where new users can test their news postings in a local environment.
And other groups that may reflect the requirements of the ISP user base, sometimes such groups are spawned from the general newsgroup where an area of interest amongst users becomes evident.

ISP Resource | ISP Resource

These pages designed and maintained by the KC webmaster. If you have any queries or comments about the KC World Wide Web pages or other services please contact us .

Page owner: <dgd@kcbbs.gen.nz>
Last modified: 7 July 1997.