IP Early Adopters: What They Learned

The first IP-PBX products started appearing in the mid-1990s, and hype around the technology grew quickly. By the late 1990s and early 2000s, pioneering telecom managers were beginning to implement IP telephony solutions, lured by promises of cost savings and amazing new applications. For organizations wrestling today with whether and when to migrate to IP, it may be instructive to know how those early adopters fared. Was IP telephony a disaster? Did the pioneers get arrows? Have they abandoned or expanded their use of IP?We delved into our own archives and those of leading IP telephony vendors and found some early adopters willing to talk about their experiences – mostly positive, we should point out, though not in all cases what one might expect.We asked them: What happened next? Were their expectations met? What have the benefits been? What were the problems? Here are their stories.Algonquin CollegeAlgonquin College in Ottawa first implemented a Cisco IP telephony system in 2000 using AVVID and CallManager products. Cisco featured Algonquin in a case study still posted on its website (http://newsroom.cisco.com/dlls/global/canada/news/2001/cp_01-01.html). The initial impetus for the college’s move to IP was the construction, starting in 1999, of two new buildings at its Ottawa campus (it also has satellite campuses in Pembroke and Perth) – the new Police and Public Safety Institute and a student residence. The college considered simply upgrading and extending the existing Nortel PBX, but sent out an RFP that also welcomed "converged" solutions.The Cisco case study doesn’t say much about why the college decided to go with IP or how it expected to benefit. The story claimed the IP system "increased efficiencies, saved time, and dramatically reduced costs." Rod Martin, manager of infrastructure services (then and now) is quoted saying the new system was much easier to maintain. It also, according to the Cisco article, paved the way for "an incredible wealth of applications," including Wi-Fi wireless phones.Today, the school uses Cisco IP telephone systems exclusively to support about 35,000 students and 3,000 employees on three campuses. Martin says Algonquin wasn’t necessarily looking for an IP telephony solution in 2000, "just a replacement system, something we could grow into." The Cisco technology met that requirement, allowing the college to grow smoothly – the number of phones has gone from 1,800 to 3,000 in five years.Algonquin was also attracted by the wealth of applications, but hasn’t implemented all those it originally planned. It was going to add online chat and instant messaging to its registration information call centre, for example, but the increased use of Web self-service reduced the need. It also liked the notion of a phone-based directory – the "killer app," Martin says – and an emergency paging system, neither of which would have been possible without IP and both of which it did implement.The paging system, InformaCast, an add-on from a company called Berbee Information Networks Corp., allows the college to broadcast emergency audio messages to IP-connected speakers around the campus and scroll text messages on IP phones in emergency areas. "That’s worked out very well," Martin says. So have the Wi-Fi phones, which are used today by service technicians and faculty in areas such as automotive technology where users tend to move around a lot.Algonquin had to make substantial upgrades to its data network to support IP voice, including switching from a single-router topology – "router on a stick," as Martin calls it – to a distributed router system. It also added quality of service (QoS) to prioritize voice traffic and multicasting to support the paging system.Those upgrades were "happening anyway," Martin now says, and having QoS on the network had the added benefit of allowing the college to replace a separate serial network for point of sale (POS) devices (in cafeterias, for example) and put POS on the IP net.Martin agrees that there have been efficiencies around reduced effort for moves, adds and changes. "Self-service is a reality." Users who move offices simply pack up their phones and plug them into the network at the new location. Training is also easier, both on using and maintaining the phones. But it’s not as if the college was paying a contractor for moves, adds and changes before the switch to IP. "I don’t think there’s necessarily a big cost savings, but we are able to offer better client service," Martin says.Other benefits? IP telephony has allowed the college to offer students voice mail access on the Web, made it easier to account for long-distance costs, and delivered much more sophisticated interactive voice response (IVR) capabilities, including true skills-based routing in call centres and departments.That Algonquin has gone all-IP for telephony is an indication that its experience has been mostly good and Martin confirms this. The IP system has been "rock solid – at least as reliable as the PBX," he says. "I can’t remember the last time we had a failure."There have been "no significant issues" around voice and connection quality either, although there were some jitter problems when IP calls first started moving to and from satellite campuses over the wide area network, which does not have QoS. Correcting it required a small change in the router software. "Since then, we haven’t had any issues," Martin says.Grand River HospitalHealthcare facilities were among the last to start the move to IP, mainly because of concerns about reliability in an environment where voice communications for front-line care givers can be a matter of life and death. In 2003, we wrote about three pioneers, including Grand River Hospital, an amalgamated entity in Waterloo County in southwestern Ontario. Grand River only looked at IP telephony in 2003 because it needed wireless phones in a new regional cancer centre. The hospital had been using Nortel Companion wireless phones for nurse call and other applications at its two main sites – the Freeport Health Centre and K-W Health Centre (495 beds total). But Nortel was phasing out Companion.The hospital already had WLAN infrastructure for data applications in the two existing facilities, so voice over WLAN made sense as a replacement for Companion. Grand River also already had Nortel Succession hybrid phone switches in place in both facilities so the organization could begin to look at wired IP voice at the same time.An initial pilot revealed some problems with the wireless phone system, which used technology from Symbol Technologies Inc. (now a Motorola Inc. company), but the trouble was not serious enough – or so the hospital thought – to prevent it going ahead. "It’s a problem that will go away over time as coverage increases," director of information systems Dwyne Patrick said at the time. These turned out to be famous last words. The wired phone trial, on the other hand, was an unqualified success.Today, Grand River has the original hybrid Nortel switches in place. It has gradually migrated to IP as phone sets wore out or operating units were moved or re-configured. The hospital installed Cat-6 cabling, which it can use for both IP and conventional digital voice phones. It does not install any new digital phones or PBX cards now, only IP. At the Freeport facility, the ratio is 80:20 in favour of IP; at K-W, it’s 70:30.Grand River does still keep some digital sets even when it upgrades a unit to IP. "We take a sort of belt-and-suspenders attitude," Patrick says. "If we’re renovating a nursing unit where we’re going all-IP, we might put in one digital set. That way, if we lost the IP card, but the digital card was still up and running, there would be at least one phone working."He could get similar redundancy by using two IP cards, he concedes, but having the familiar digital phone as back-up gives the organization an extra measure of comfort.The wireless phones were a minor disaster. "They were terrible," Patrick says of the first Symbol phones and access points. "You could have a good conversation if you stood still under the access point (AP), but if you wanted to walk, forget it."The problem was more fundamental than he initially thought (and we reported). The vendor had in fact misrepresented the system’s ability to enable roaming. The phones latched on to the nearest AP and continued to try and communicate through it even when the user had moved down the hall past two or three other APs. "So of course voice and connection quality degraded," Patrick says.Grand River continued to use the Symbol phones for a time, but it’s now deploying Nortel phones (Nortel sources the phones from SpectraLink Corp). It also switched a year and a half ago from a traditional fat-AP wireless network architecture to using access ports in some facilities. The new wireless infrastructure makes it easier to control and configure the roaming function. "Roaming is much, much better now," Patrick says. And it should continue to improve as the hospital switches to access ports everywhere and fine tunes the network.The wired IP phone deployment, aside from a few early "hiccups," went much better. "IP is indistinguishable from digital in terms of up time and reliability," Patrick says. "And Succession seems to be as stable as or more stable than its predecessor."There were two occasions on which the QoS settings on one of the Passport data controllers went "a little offbeat," resulting in some echo on the IP phones. It wasn’t enough to stop people using the phones and the problem was permanently solved with a minor software upgrade the second time it occurred.There have been few complaints about voice and connection quality on the desktop sets. Patrick made sure that the users most likely to notice and complain – including the hospital’s top executives – were equipped with IP phones. "Some of them think they can tell a difference but it’s not enough to bother them," he says. "My bet is that I could switch them back to digital and they wouldn’t notice."Asked about the benefits of going IP, Patrick first notes that the hospital didn’t have much choice because the vendors are all moving to IP. Grand River has not implemented many, if any, advanced IP voice applications either – it had already implemented a phone-based directory using pre-IP technology, for example – so there was little return there.However, he notes that "it’s less expensive to maintain one architecture." And some of the promised savings around moves, adds and changes have materialized. In the past, when a user moved, the hospital sent a contract telecom technician who had to visit old and new locations and at least one wiring closet. Now users move themselves. Patrick is using that contractor less. The savings amount to $500 a week.In the case of new installs, he can send any of his network technicians if needed; in some facilities, the security guard or the head nurse plugs the phones in. Any issues that arise are addressed remotely.Sudbury & District Health UnitThe Sudbury & District Health Unit (SDHU), the public health agency for Sudbury and Manitoulin in northern Ontario, made the move to IP voice even earlier than Grand River, in 2001. The SDHU was looking for a new phone system because its aging and often-upgraded conventional PBX was running out of steam and the organization had undertaken a major renovation of its main facility in Sudbury. It ended up choosing a Cisco AVVID solution and, as in the case of Algonquin College, Cisco published a case study on its website, which is still there (http://newsroom.cisco.com/dlls/global/canada/news/2001/pr_10-30.html).The agency has four offices in Sudbury interlinked over a municipal fibre network run by Greater Sudbury Telecommunications Inc. It also has four branch offices outside the city, two of which are connected to the main telecom facilities in Sudbury over broadband cable connections.SDHU’s senior network analyst, Craig Tyers was responsible for choosing the new telephone system in 2001. He told Cisco at the time that the system he was looking for had to be flexible and scalable to change with the Health Unit’s business. It also had to be easy to maintain, and it had to offer a very clear return on investment (ROI) with rock-solid reliability. That’s exactly what he got, Tyers says.Total cost: about $150,000. That includes network infrastructure upgrades – which would have been done anyway – that are for the most part still in place five years later. The agency implemented power over Ethernet (PoE) specifically to support the phone system, to ensure it didn’t lose connectivity in a power outage. "We just can’t afford to be out of business during outages," Tyers says. The CallManager and some other servers on the network have since been upgraded as a matter of routine maintenance.The ease-of-maintenance part of Tyers’ system requirements delivered most of the hard-dollar ROI. The move to IP eliminated the need to call in suppliers to do moves, adds and changes. "During the renovation, we moved every staff member four to five times," Tyers says. "It doesn’t take long to calculate what external suppliers would charge to make all those changes."The savings on moves, adds and changes during renovations covered all hardware costs associated with the new phone system, he says. He knows of agencies going through amalgamations and other changes that blew entire annual telecom budgets on moves, adds and changes because they stuck with traditional phone systems. Tyers estimates that the SDHU continues to save about $10,000 to $15,000 each year.There were also savings on wiring. The agency had to reinstall all wiring in the renovated facilities. "A person could realize a 50% savings in an environment where gutted the entire building," Tyers says. "No more traditional telephone wires."The agency now has greater control of its own environment. In the past, it sometimes had to wait two weeks for a local vendor to make needed changes. Now it can do virtually everything using internal resources. "We had to be able to react more quickly to business needs," Tyers says, "and that’s what VoIP gave us."One other key benefit: caps on numbers of phones supported by a switch are much higher with IP than traditional PBX products. This was important to the SDHU since its staff strength has grown from 180 to almost 300 in five years.The agency’s approach has been to keep the IP phone system "simple and clean." It has implemented no advanced applications, although it is now beginning to install wireless phones and is experimenting with PC-based softphones.Because the outgoing Mitel PBX was at the end of its useful life and was experiencing routing errors, reliability with the new Cisco IP system has been markedly better. "There has been virtually no down time," Tyers says. "It’s been rock solid." The phone system has stayed up even when the agency was doing routine updates and connecting new outside trunk lines.As for voice and connection quality, Tyers concedes there is "an art to doing noise cancellation" and each one of the Cisco gateways the SDHU has used required some new learning. "You’ve definitely got to know what you’re doing," he says. When tuned properly, the phone system has fewer noise problems than the old PBX.ConclusionGrand River Hospital charted a more conservative migration to IP than Algonquin College and SDHU, and it experienced more challenges – not that there appears to be any connection between those two facts. None of our early-adopters faced major setbacks, and all are moving rapidly towards or have already arrived at a pure-IP strategy. This is not to say that there are no challenges involved in implementing IP phone systems. Tyers’ cautions about the need to learn "little tricks" for fine tuning systems in order to avoid noise and echo problems and Grand River’s early experiences with voice over Wi-Fi are evidence of that. But on the whole, our pioneers say they did not get arrows. Then again, maybe they’ve just forgotten the pain.

IP Early Adopters: What They Learned

News | 02/01/2007 5:00 am EST

To keep reading, get a free trial.

Already a Subscriber?

Reuse & Permissions

Inside The Wire Report

Information

Hill Times Publishing

Get in Touch With Us