Dimensioning an Asterisk system
Typical questions asked on the mailing asterisk-users are:
How fast/big must my machine be in order to serve my needs?
How many simultaneous calls can Asterisk handle?
Unfortunately there are no simple answers. You'll need work through the following checklist to at least get nearer to an answer or be able to post a meaningful question to asterisk-users:
1. What type of phones do you plan to use (analog, SIP, Skinny, H.323, MGCP)?
2. How many phones will you operate?
3. How many external lines do you have, and of which type (analog, BRI, PRI, T1, VoIP)?
4. How many concurrent internal/external calls do you expect (example: 30%)? Consult the Erlang tables (see links below) if in doubt.
5. What codecs will be employed, and will you need to do a lot of transcoding? Tip: Enter SHOW TRANSLATION at the CLI.
6. What features shall the system provide (echo cancellation, voicemail, conferencing queues & call center, recording, fax, voice menu, text-to-speech, speech recognition)?
7. How reliable/scalable must your system be?
8. How many Asterisk boxes do you plan to put in place?
9. What does your IP network look like (speed, QoS support, VLAN, Power-over-Ethernet)?
The next steps are to a) check out the hardware examples (use cases) section of this Wiki, and b) after that possibly set up a test system that is able to simulate your requirements so that you can run a load test e.g. using .call files generated by a script.
How many concurrent calls?
Here a collection of statements voiced on asterisk-users:
Note: When adding to this list, please provide CPU & Memory usage, not load average. These are much more concrete metrics and have much greater relevance when comparing different hardware and operating systems. Entries are sorted by CPU size.
* for old & slow CPUs take a look at Asterisk hardware recommendations
* Gumstix-based SIP-to-IAX proxy server can route about 40 calls concurrently (no transcoding, Gumstix 400XM, voicemail on CF card)
* Pentium 133 MHz, 16 megs of ram: Handles up to 3 concurrent SIP calls before quality degrades (no info about transcoding)
* Linksys NSLU2 aka "slug" (NAS Device for $79): Manages around 4 SIP calls with g711 or gsm (but not g729) on its IXP400 processor
* Pentium 1, 166mhz, 32meg ram: Successfully runs 4 SIP calls with codec g711
* Pentium II 233/64 RAM: 2xBRI (=4 ISDN channels) plus a lot of SIP devices
* The SC1100 CPU on the Net4801 will only successfully encode two streams of voice using G.729a compression. It is generally accepted that Asterisk requires about 30 MHz of CPU power per active voice channel. Thus the 266 MHz CPU on the Net4801, in theory, supports around eight simultaneous calls. This presumes that all calls are G.711 encoded audio. The Soekris Net4801, while inexpensive, has been absolutely reliable as a host platform, delivering adequate processing power to support six extensions and up to four active lines at one time. It could perhaps do more, but such was beyond the scope of my ability to test.
* (Mai 05) on a Soekris board: purely translating from G.729 IAX2 <-> G.729 SIP (maybe using IAX trunking), note that's not a codec translation: I have people doing at least ~30 calls in that manner (using AstLinux)
* A Soekris 4801 can easily support 20 - 25 SIP clients if they are all running the same codec (I use ulaw), I know of others that have 20+ sip clients and a t-1 card in the soekris for zap channels and it works fine. If you have to do any sort of transcoding a soekris is not the way to go but for a small installation it works great.
* Soekris 4801 at 266 MHz: Four concurrent ZAP calls might be tricky: even if you aren't doing transcoding, you will still probably have to do echo cancellation, which is CPU intensive (especially on the Soekris)! As far as 60 calls: maybe SIP to SIP, with re-invites and even that is pushing it.
* Capacity of MeetMe: With 28 persons on a Pentium II, 300MHz, 128 MB vmstat shows 70% idle.
* 5 calls to music on hold (where it's transcoding from the GSM moh file to G711 is causing my 533MHz VIA processor with 64Kb cache is using between 5 and 12% CPU.
* "I have anywhere from 15, to a peak max of 30 traders all using the same meetme conf during the day. My * is running on a old 4U 500Mhz machine (dual board, one processor installed now). With the exception of of a few problems from software sip phones, our implementation has been relatively problem free."
* VIA embedded 800 MHz: Max. concurrent and transcoded calls with speex: 4, ilbc: 9; g729: 11, g726: 22, lpc10: 24, gsm: 25, A-law: 83
* VIA EPIA 1GHz fanless: Compressed codecs are fine - as long as you aren't transcoding ;-) I figured I could push 30 non transcoded calls through one, but I've never had the ability to fully test it out. The max. I had going on one system was 20 calls. This box has 256MB of RAM. Asterisk 1.2.16, zaptel 1.2.15. It's running apache+php, sendmail, and asterisk. Boot it off flash and have it load an initrd.gz into RAM. Everything will run entirely from RAM - no writes to the flash at all! I can get everything inside a 48MB flash drive
* Asterisk 1.2 beta managed pretty well at SIPit: I ran Asterisk on a 1 GHz laptop and could easily handle 250 concurrent calls with re-invited RTP, That took about 9% of the CPU. With RTP codec conversion (ulaw-alaw, easy load) my poor laptop run at 99% load already around 50 calls...
* Celeron 1GHz with 2 Gigs of RAM that handles 72 DIDs, 24 connections to our traditional PBX, and is our incoming FX server. (Also running 1.0.9)
* Dual P3 1.13ghz, with 5 PSTN Channels via PRI -> G729 channels, has a constant load of 0.45
* I do have a dual 1.266 PIII with 2 GB of RAM that handles 75-85 concurrent SIP (GSM) sessions + 3 IAX2 trunks for a total of about 100 calls at the same time (using * 1.0.9). I also have this server set up for a file server (authentication using LDAP in a Win2K ADS domain).
* Intel P4 1.7G 512mg 20 gig HDD with 2 T400P boards for access bank and 60 analog phones; 24 channel PRI
* (Apr '03) chan_h323: Takes advantage of the codec pass-through feature of Asterisk (unlike chan_oh323), but has no jitter buffer. We simulated 125 concurrent calls setup in random (sub-15 second) intervals, using passthru, on a 500mhz P4 without causing more than a 5 second delay in call setup time. On that same machine we simulated 45 concurrent calls using GSM transcoding, again without a long delay in call setup time. We have also a few production systems (single P4 2.53 ghz, 2gig ram) in pass-thru mode with 3 T-1s of concurrent real traffic.
* Today, we were able to reach an astonishing 790 simultaneous audio playbacks. (sip to asterisk, any codec.) on a 350$ pc. Tests were done on we a pIV on 2.4ghz with 512mb ram. Unless you dont do cpu hungry tasks such as transcoding you can go as low as 1% idle cpu cycles without the slightest audio glitch in this test
* Celeron 2.4 GHz, 386 MB RAM, 1xT1 for (only) 10 phones
* 2.6 GHz Pentium4 800MHz bus w/ HyperThreading enabled, 2 GB RAM (however 1 GB really is sufficient): 40 concurrent SIP-to-ZAP calls (maybe more are possible): "Measured for about a 12 hour period and over 5000 total phone calls per day. With this server you can see the load average jump from 0.00 to 6.25 in a matter of a minute"
* g729 performance taken from the Digium web site: Internal testing with dual Intel� Xeon 1.8GHz processors allowed 60 concurrent calls. Dual Xeon 2.8GHz processors allowed 80 concurrent calls.
* "The load is extremely depandant upon what you are doing with it. For example, a simple IVR/Zap-T1-channels-only system can handle 10 times the number of consecutive calls of a SIP&Zap conference call system (at least in my experience)."
* MeetMe configs on single P4 (512M and 2G). I've seen no technical problems up to and over 30 attendees, all coming in via H.323 (chan_h323), using g.711u (no transcoding). Word of warning- above 10 people, a single misconfigured softphone attendee can create chaos. Based on load averages (at 25 people, single P4 with 2G ram and nothing else going on on the box, we see about a 0.5 load average), there's no reason why it wouldn't support 50 or 60 attendees before seeing problems.
* P4-3Ghz/1Gb RAM: Native format_mp3 for MoH running on FreeBSD 5.4 with Asterisk CVS HEAD (July 2005). mp3 decode quality is great... up to 70 simultaneous MoH (8khz 16bit mono mp3) calls with about 80% idle.
* (Aug 05) Pentium 4 (No HT) 3.0Ghz, 1 GB Ram. Handles 46 G.711 SIP calls while playing SayDigits(1234567890) to them all. Load average 0.43.
* We've been doing some benchmark experiments on a 3GHz HT box with 1GB of ram, mirrored traditional IDE disks. The box has a Digium quad-PRI, a TDM40B, a TDM22B and a Sirrix quad-BRI board in it. This box can run 120 active calls over 4 PRI spans. Its running MusicOnHold into 60 of the channels, playing various GSM prompts into the other 60. The "user" cpu usage is about 25%, the "system" cpu about 25% also. (I imagine the system cpu will go down if I remove the "extra" boards). I can add to that 5000 registered SIP peers and 5000 registered IAX2 peers - total of about 100 registration refreshes per second. That adds about 40% more user CPU and pretty much fills up CPU. Audio quality is still perfectly fine though, and PRI slips few and far between. Load average for the whole mix sits between 5 and 10. About 550Kb/s out and 400Kb/s out on the ethernet for the registration traffic.
* 3.2 Gig Pentium 4, HyperThreading (HT) turned off, Memory 1 Gig, no hard drive, running in Ramdrive, Memory usage never goes over 256Meg. I'm using asterisk with looping call test configs to play audio and using 3 of the same spec servers to pound calls through 1 server. I managed to get 350 concurrent calls through (g711, no transcoding) with perfect audio consistently with ~20% idle processor load. Anything above that and things start breaking up. Using Asterisk 1.2.6 I'm running into a limit of 276 SIP calls and no more. IAX calls can go 400+, so I test with combination 200+ SIP calls and the rest IAX and a combination of more and less SIP and IAX calls. With HT turned on, SMB loaded in the kernel gave ~20% performance increase, BUT, using 425+ channels gave very inconsistent results, choppy audio, calls dropped, no audio, and call setup time slowed.
* "Dual Xeon 3.06, 120 channels (60 calls) with H.323, G.729 using less than 10% CPU, no transcoding is being performed"
* We installed a quad xeon 3ghz which transcoded ~100 active channels (as a gateway). Take a look at the codec demands (in asterisk show codecs I believe) and scale from there. This box was 60% loaded - which is all we're comfortable with before latency goes too high.
* "Look for the recent 'capacity testing' thread. We've had some discussions on it, but so far the bottom line sounds like you won't be able to run more than 20-25 decent quality calls before asterisk dies when transcoding and H323 are involved." (note: maybe this applies to oh323? For h323 the tests shown at Astertest indicate that for high CPU codecs like g729, ilbc and speex the protocol doesn't matter, so h323, sip and iax should all perform the same here)
* "I read somewhere that one of our associates here has experienced about 45 calls when transcoding. The question becomes if this is all we can do with one server, what is the point of getting 4 E1s Digium card while one can never be able to use 120 channels transcoding from VOIP to TDM?"
* "My rule of thumb has been 100 G.729 channels for a dual 3.0ghz Xeon machine. Cost: around $42 per channel (I buy SCSI systems with RAID, and that includes the $10 charge for the G.729 license.) (Not including Digium 4 T1/E1 card) Your mileage may vary in both performance and price. Cost for an AS5300-style solution: around $110 per channel, and that's on the used market pricing plan."
* On my own testing I could get 200 simultaneous calls on a Xeon 2.4 using SIPP, each call was 10sec so the box had 20 incomming cps, obviously this was using SIP not Zap. Since I don't have another Xeon to put on my box how do MP scale ? I believe that I cannot get 400 calls for 2 Xeon but maybe 350.
* (Jan '06) Dual Xeon 2.8 GHz Intel Chassis with 1GB RAM: Running +-35 IAX2 channels out a 2 x PRI (E1) Digium Zap channels. Importantly the box is recording all the calls and logging all cdr's in a mysql DB.
* (Apr '05) Dell PowerEdge dual 3.2 GHZ XEON (32 bit), 2GB ram: It's configured using realtime-extensions / sipfriends / iaxfriends with a local mysql daemon, 80% of all calls are IAX <-> SIP calls with no codec transcoding and no jitterbuffering, and 20% of all calls are IAX <-> IAX native transferrred calls, again no codec translation. The box has no zap channels. Now, when I have a load of 100 simultaneous users calling (that means I have 200 simultaneous lines open of IAX and SIP calls (where 80% of those calls are IAX calls translated to SIP/RTP) then asterisk uses an average of about 30% to 35% processortime (divided over the processors) and mysqld then using about 1% average load .. so the maximum callers I can have is about 300 ..
* One very important thing to remember when you are talking about SIP-to-Zap calls in Asterisk is the processor spikes. Taken on average, a quad processor machine might be able to handle 200 concurrent Sip-to-Zap conversations, but the reality is that there is a 4x load spike that occurs at random times with Sip-to-Zap conversations that would kill your big box when one occurs.
* An Asterisk system can only handle a max. of 250 concurrent ZAP channels. This is due to the design limit (255) within the ZAP channel driver. __Update Sep 2005: That limit has been removed a LONG time ago. Zaptel provides another interface for opening device files past the ~250 or so that exist in /dev/zap.
* Mark Spencer on Mon, 2004-11-29: I've got IAXTel moved over to "realtime" but it's still very overloaded apparently causing long delays. There are currently 6000 registered users, over 1000 of which are typically actually registered to the iaxtel box at one time.
* (Oct 06) I am doing testing with SIPP tool. Now the thing is that asterisk is not able to handle more then 550 concurrent calls with call rate 100 call/sec (on my amd 64bit dual core machine with 2 gb ram). I am doing this with 'canreinvite=yes' mode so there is no issue of RTP traffic. I just want to astablish peer to peer call thru asterisk (I hang up calls after some time so this is including all invite and hangup related massages)
* (May 05) In my testing of CVS-HEAD I can get 5551+ sip "sessions" without media on asterisk without a problem (testing with only 1 SIP user). The load average is around 2-3. On a side note... on my 3ghz P4 HT box I can get 629 ulaw sip calls with media (verified) without a problem. The load average on the box was around 14 and it still sounded perfect... so if you had a dual 3.4 ghz Xeon box you should have ZERO problems doing a DS3 with asterisk. (That is if the interrupt is 1000 per second and not 28000 and its all ulaw) ... note that if you do not set the ulimit -n 100000 or something similar efore you start asterisk you'll run out of FD's around 151 calls.
* (May 05) But the real scalability wall I've seen is number of registered peers... That's what takes down a box (at least with IAX). I've heard reports of a Dual Xeon 3.2GHz not being able to handle even 1000 IAX peers. ... I managed to get about 2500 users online on one box by modifying iax2.h (reg expire changed from from 60 to 240 seconds) decent IAX clients will comply with that setting .. b.t.w. don't go higher, because many NAT gateways will close their dynamic NAT mappings after 300 secs, a few some even after 30 secs!
* (May 05) While you may not choose to put 10k users on Asterisk, I have. Many more, as a matter of fact. Some of these systems were simply media/application servers, while some handled registrations as well. While I agree that Asterisk needs some help on registration volumes and scaling, I'd not sell it short so quickly. At the moment, the only reason I still would use SER would be for the registration and call processing/loadbalance speed - Asterisk provides all that I need for back-end call processing.
* (Apr 05) Signate Telephony Server 5000�s 51 Gigabits per second I/O capacity sustains more than 5,000 Session Initiation Protocol (SIP) call streams per module using 80% of the capacity of a gigabit network. Up to eight uniquely scalable modules can share a 6.4GB/second interconnect based on SGI technology that enables a coordinated system supporting over 40,000 simultaneous call streams.
* CTC Connections (www.ctcconnections.com) provides an full features Multi-Tenant asterisk cluster system with load balancing, high availability, DNS SRV with ability to remove a server from the cluster at anytime without informing the cluster management system. It scales upwards of 20,000 + calls with media pass trough and 100,000+ with no media pass through.
Providers
VON magazine Dec. 2005: For 2004 Voicepulse announced having run 10 million minutes of IAX traffic. In Dec. 2005 that is about the montly amount of their IAX traffic. An Asterisk-based ITSP in Brasil claims 100.000 users.
Call centers
* (Nov 05) We use boxes for outbound dial setup. I don't think I'd attempt to go past 4 E1s (120 lines) which would be 5 T1s. If the box is running hard like that the load average will sit around 7 or so, still fair amount of spare CPU but there is no way an Asterisk box will run well with the CPU anything like maxed out. Our site has 250 agents or so and the work is currently spread over 6 servers with 3 E1 PRIs on each. Each box makes around 3000 to 5000 call attempts per hour.
CPU load graphs
On this link http://quasar.sourceforge.net/bench.htm you can find the article from which you can obtain CPU loading depending on number of immediate zap-to-sip calls information.
Lessons learned:
* with zaptel software echo cancellation switched on the number of calls a box can take is cut in half (measured with transcoding SIP <--> E1)
* without echo cancellation up to 240 concurrent SIP-to-E1 calls are estimated on a 3 GHz Debian system with 2.6 kernel
Determining limits
What really matters is the load on your system and the type of channels you are using. Zap channels are more reliable at extremely high loads than any VOIP channel. Manager Actions can fail at extremely high loads. All VOIP channels will suffer quality loss at extremely high loads. By extremely high loads I mean about 5.00 and higher of system load (10.00 on dual CPU machine) There are other factors, but this has proven fairly reliable for us. System Load by channels in use is different from server to server depending on the CPU/RAM/Motherboard and other application running on the server.
Digium Hardware (July 2005)
Digium cards are now smarter! Digium is proud to announce second-generation firmware for Digium multispan TE-series hardware PCI cards. This firmware provides several new features. These features allow Digium cards to do more work in hardware and leaving the server's CPU open to do other tasks, all of which contribute to less CPU overhead, and therefore improved performance, and more channels running through a single PC or server.
... This is a 67% increase over previous benchmarks ...
The new firmware solution relies less on the server's CPU for the operation of the Digium card, therefore reducing CPU overhead and improving performance and increasing channel processing. For example, a dual-processor, 3-GHz 800FSB Intel XEON server with 1MB L2 cache, and a Digium 4-port T1/E1 card, can now convert 120 SIP channels with G.729 compression to the PSTN without Digium's echo cancellation module and 150 channels with G.729 compression with the Digium echo cancellation module.
If you have very high traffic volume
Scott Stingel www.evtmedia.com writes:
My largest client has a configuration (15 asterisk boxes) that takes up to 1800 simultaneous IVR calls, supplied by a large private DMS-100 PBX over 60 E1 spans. The calls typically are very short (5 seconds).
* I suggest: Use 1 processor (chassis) (example: 2.8Ghz P4) for every 4 E1's (example: one TE405P card) when you have a large amount of call setup traffic. I tried a dual-Xeon to see if I could handle two TE410P's instead of one in a pure IVR (Zap) environment, but many calls were dropped under full load. So I've now decided just to put one TE410 or TE405 in a box as a maximum. If you are doing a limited amount of transcoding, and don't have a lot of call setup load, you might get more than 120 channels in a system, but be cautious! Also, I haven't re-tested using the new Digium firmware, see above)
* Try to do as much as possible using the dialplan (extensions.conf). AGI scripts are very powerful but cost you in performance when you're running large numbers of lines. (more info, see: designers' FAQ #2)
* For the processor to use, see the Wiki for suggestions. I've used P4 and Xeon based Tyan and Intel motherboards with success.
512 simultaneous SIP-to-SIP calls with Digital Recording How to use a RAM disk to eliminate the I/O bottleneck associated with digitally recording calls via the Monitor application Hardware for Asterisk: Quad Xeon MP CPU at 3.16 GHz with 20 GB RAM (Dell Poweredge 6850) The 512 calls was also the only load on the system during testing. We are actually using the box in production now and with the additional load of agents, announcements, MOH, unattended monitoring, NFS transfers of the leg files to a remote box for mixing, etc. it looks like a more realistic number is somewhere between 250-300 simultaneous calls. For instance, at this moment we have 45 calls in queue and 116 active calls (resulting in roughly 250 leg files being generated) and the box is about 65% idle. I'm assuming that the scaling will be somewhere between linear and exponential, which is where I'm deriving my projected numbers from.
* Original posting
* Follow-Up
Load testing - Zap
Scott Stingel: You can build Zap load testers for asterisk, using the outgoing call facility (see sample.call). What I've found:
(a) I don't think it manages queuing very well if there are a limited amount of outbound resources. For example, if you define a group ("g9" for example) of two lines for use in outbound calling, it works fine if the number of outbound calls to be made at any moment never exceeds 2. A third call file in this example, will be grabbed by asterisk, but will fail immediately. So I had to create a mechanism in my Perl script to ensure that the outbound calls actually completed - no easy feat since I couldn't tell when that occurs from the perl script too easily.
(b) There was a problem dumping more than about 12-15 outbound calls at once in the outgoing directory, even if there were plenty of channels available to make the calls. Asterisk would grab them but would not process some of them. This is a load-testing scenario, and not too common I realise, but something to be aware of. It didn't seem to matter if I switched to a more powerful processor. Note that this problem may have been fixed (see below).
Jesse Janzer: Just a quick mention, we implemented a generic paging system and regularly dump more than 30 files into the outgoing spool (during an agi script) at once to accomplish the page, never had a problem...
The outgoing facility works fine at lower call volumes, if you stagger the creation of the files in the outgoing directory.
Tip: You can manage when the call starts by setting the mtime of the blah.call file. See the sample script at http://www.fnords.org/~eric/asterisk
Quad T1 card performance testing - Digium vs. Sangoma
Load Testing - IP
Juanjo: Remember to increase the max number of file descriptors and RTP ports.
Using SIPP
Test1:
I have reached 1500 concurrent SIP/GSM calls against the Echo application at 50cps of 10s duration on a Xeon 2.4Ghz single cpu. Anyway I�m trying to make a more real scenario since I believe Echo application doesn�t process audio too much just copies it.
When testing I monitor Asterisk against a 7960 calling MusicOnHold at about 500-600 concurrent calls I start to have audio dropouts.
Test2:
I set up a UAC which was generating 10cps of 10s duration and the corresponding UAS which received this calls. The command used to generate the calls which were GSM was:
sipp 192.168.65.100 -s 700 -sf uac.xml -d 10000 -r 10
The command to receive the calls on another box was:
sipp -sf uas.xml
I�m using my own uac.xml and uas.xml just to talk GSM, I monitored using my 7960 against a MusicOnHold. On my Xeon 2.4Ghz no call were dropped and no audio problems. Note that I use nat=yes and canreinvite=no for UAC/UAS in Asterisk config sip.conf. Older versions of SIPP didn�t support authentication.
* For 40cps of 10sec duration (which means 400 concurrent calls) it works just fine for me.
* At 50cps of 10sec duration no call are dropped but I start seeing some SIP packets retransmitions.
* At 60cps lots of call gets dropped but the funny thing is that the audio through the 7960 isn�t much affected.
However: I'm not seeing any RTP traffic generated from SIPP UAC which means that * won't generate any traffic at all since it recover the clock from the received stream AFAIK. So maybe the tests aren't measuring the whole * capacity just the SIP stack. One solution is to end the calls on a MusicOnHold extension but passing through an intermediate * which is the measured one.
A cleaner solution for Asterisk 1.2, however, is outlined here: Using SIPP to stress-test Asterisk. Finally bug/patch 5374 should fix this for Asterisk 1.4 and later.
SIP testing (May 2007)
I've drawn three conclusions from this set of benchmarks (uLaw only, no transcoding, Asterisk business edition: ABE-B.1-3).
1. At low call volumes, the dual-core server outperforms the single-core server by the expected margin.
2. Calls bridged to an agent are more CPU intensive than calls listening to audio via the Playback() application or calls in queue. This is expected, because they involve more SIP channels and more work is done on the RTP frames (bridging, recording, etc.).
3. For all call types, the majority of the CPU time is spent in the kernel (servicing system calls, etc.). I've observed this to be true at all call volumes on our production server, with the ratio sometimes in the range of 20 to 1. This may suggest that the popular perception that Asterisk doesn't scale well because of its extensive use of linked lists doesn't tell the whole story.
Notes
* Transcoding is very resource hungry (CPU intensive); the same applies to echo cancellation
* g729 codecs licenses can be purchased for Asterisk, whereas g723.1 licenses are not available and therefore Asterisk can only provide pass-thru functionality for g723.1
* IAX trunking can help to significantly reduce network traffic for 3 or more simultaneous calls between two Asterisk machines
* Hypterthreading is not well implemented in the 2.4.x kernels; the newer 2.6.x versions are more advisable to be used if HT is desired
Erlang tables
Calculating the number of external phone lines needed
* http://www.erlang.com/calculator/
* http://personal.telefonica.terra.es/web/vr/erlang/eng/mtaula.htm
* http://www.itu.int/itudoc/itu-d/dept/psp/ssb/planitu/plandoc/erlangt.html
* http://www.ct-labs.com/Dr%20C/q27.htm
Entradas
