Our Call Waiting Tone Went Missing

A good part of the last couple days has been spent looking for missing call waiting tones. Call waiting tones are the soft background beeps that usually are heard when you are talking on the phone and someone else tries to call your phone. The sound you’ll typically hear is a double beep of around 440 Hertz for 300 milliseconds. Well, we lost ours.

We’re not entirely certain when we lost it. Looking back, it appears that the first subscriber report may have been received in mid-September. Only recently have the volume of complaints risen to a level that caused anyone to intervene. (There’s probably a process improvement or two lurking within that statement.)
Because call waiting tones are provided by a component of the Nortel switch, we immediately opened a dialog with their Technical Assistance Service (TAS.) Starting yesterday morning several engineers at Nortel TAS with on a conference bridge with several of our Engineers trying to isolate the problem.

But in the meantime the real story is happening elsewhere. Apparently, once enough people became informed and excited about this topic, then everyone became excited about this topic. Is this the 100th Monkey Effect? Some people were asking for hour-by-hour reports, others said send an email to the whole company. C’mon people, it’s call waiting tone, for crying out loud. Some people so easily lose sight of their priorities.

I tell you that I worry that if there is all this drama caused by call waiting tone then I should consider myself damn lucky the most recent OTA outage didn’t last any longer. Where will it end?

Back to the engineers troubleshooting. Let’s talk a bit more about how these tones are created. The sound actually lives as a simple sound file on the hard disk of one of the computers that comprises the Nortel switch. The file gets loaded into memory of the computer and that memory is played whenever requested.

Only it hasn’t been. While troubleshooting we’ve determined that the computer really is trying to play the sound because a subtle change occurs in the background noise when on one call and a second call comes in. So the team did extensive testing to determine where the sound was getting lost.

Late this afternoon they thought they had localized the source of the problem to a possibly corrupted executable or configuration on the gateway controller. Nortel suggested that by switching to an alternate executable the problem would be cleared. They’ll begin testing that theory in a couple hours during the local low-traffic maintenance window.

I written elsewhere about our ongoing migration from MTX13 to MTX14. Although the root cause of this issue with the call waiting tone is yet to be determined, already there is suspicion that an earlier step in the migration process introduced this problem, and that it has gone undetected until now.