Can you troubleshoot phone problems remotely?

Last month I got a call from a technician at a federal government agency that has modems all over the country that talk to his main computer on the east coast. He had a problem connecting or staying connected to some of them.

The technician wanted to buy an "amplifier" to put on the modems at the main computer. He figured that if he made the sound louder at the main location all the modems would work better. Uh, no.

I guess he's used to troubleshooting computer problems remotely. Troubleshooting problems on analog phone lines with analog phone equipment (like a modem) is totally different than troubleshooting a digital problem.

When I told him that he's going to have to pick just one site with a problem, figure out what's wrong, fix that one and move on to the next one, he said "We can't do that."

He said the modems are spread out all over the place and he can't send technicians out to fix the problems. He wanted to do one thing at the main location to fix all his problems.

I got a call about the same time from a radio station who said their on-air callers sound distorted so they wanted to make the lines quieter (attenuate them) to prevent the distortion. I suppose that's an interesting idea but without checking the phone lines themselves with a butt-set or checking voltage, current and circuit loss, which he didn't do, that's a heck of an assumption and would probably leave the guy spending money on stuff he doesn't need and chasing his tail for quite a while. He wanted a magic gizmo that would solve his problems.

The previous Sunday evening I had watched a segment on "60 Minutes" about US nuclear missile sites (ICBMs). In an interview a soldier at the site mentioned that they had problems with their phones. In a later interview Major General Jack Weinstein said their telephones were analog. That it would be years before they replaced them with digital phones (for billions of dollars).

Well, all phones are analog to some extent because our ears and mouths are analog. Although someone might be able to figure out how to put a USB connector in our necks someday, I don't think it's coming anytime soon. Most phones even use our analog eyes to see flashing lights or displays.

Analog phone equipment and POTS lines have been repaired every day for over a century. It's not nuclear science and almost anybody can fix phone stuff with a little experience.

Then I realized that the missile silos are all located in hard to get to parts of the country. Even though broken phone stuff is easy to fix, the logistics of getting someone who understands a little about phones out to those sites isn't easy.

But there is absolutely no choice. Phones don't fix themselves. You can't remotely fix wires and replace worn parts (like a handset, cord or hookswitch). Same problem the guy from the other government agency has. They don't have the manpower to send out to the remote sites, and the guys who they do have are likely to be computer guys who don't understand phone stuff.

The worst part for both of these government agencies is that they've already waited until there are lots of problems. If something is maintained in a reasonable matter, it's just a service call now and then. When someone lets the problems build-up with any kind of gizmo, even a car, fixing all the problems is overwhelming and a new device may be in order.

If there's no new device available, the technicians tasked with maintaining the equipment are overwhelmed and stressed. Stressing out employees is a good way of losing them. When you lose a good employee, you lose all the knowledge they have of your particular products and customers.

The military has a bigger problem in that all of their workers (soldiers) are contract workers. At the end of their contract (tour of duty) they move on for greener pastures. Along with all the knowledge of the equipment they've been working on and the specific sites they've been maintaining.

In businesses, the bosses often tell the IT guy that he has to fix the phones from now on (to save money on the expensive telecommunications vendor they've been using). The boss figures phones have wires just like computers, and the IT guy is sharp. Any maintenance contracts get cancelled. That often doesn't work out too well since the IT guy doesn't know anything about phones and he doesn't want to look bad to his boss by calling in the old (expensive) telecom vendor for help. The real loser in the end is the boss and the company.

I've always wondered how many bosses tell the IT guy to fix their $60,000 Mercedes? After all, it's got wires.

Digital stuff is 1s and 0s. Usually +5VDC or 0VDC. On or off. Put the ones and zeros in one end and they come out the other end rearranged in the way the computer in-between wants to arrange them.

Analog stuff is totally different. A modem or fax is just like our voice. One end makes beeps and boops at particular frequencies and the other ends hears them and replies with similar sounds. If every fax or modem was connected to exactly the same central office, with exactly the same number of feet between the central office and fax or modem, the fixes would be pretty much the same for every case of trouble.

BUT:

• There are several makes and models of central offices, with different line cards. The different line cards look different electrically to the telephone equipment at the subscriber end - and they can sometimes be optioned for more (or less) loop current (including mistakenly optioned for one or the other).

• There is a different amount of wire between every subscriber and every central office, and the wire could be bigger or smaller.

• There could be load coils either properly or improperly placed on the wires (load coils are inductors used to raise the volume on a long phone line, but it often kills data being send on the line).

• There may be pair gain equipment between the central office and the neighborhood the fax or modem is in. Like a T1 or fiber. The line card that faces the subscriber is different for every brand/model of pair gain out there and can sometimes be optioned to look different electronically.

• The phone equipment's impedance may not match the impedance of the CO or pair gain line card, causing really strange audio problems (and faxes and modems send audio beeps and boops to the other end). The impedance can sometimes be optioned (including mistakenly optioned).

If you figure in all of the above variables, you can see that no two phone lines and no two makes or models of phone equipment looks alike.

The design of the telephone equipment attached to the line may be good or bad. Almost every piece of phone equipment uses a different circuit and different components facing the phone line.

Sometimes the engineer designing the equipment chose the right components that would work in almost every condition (which was the case in the old days), but more often these days the cheapest possible components were used making it more likely the equipment won't work right on any particular phone line.

It takes a technician with a spare piece of phone equipment (for troubleshooting), a meter and butt-set to diagnose why a piece of phone equipment/phone line combination isn't working right. Can it be done remotely? No.

Here's what a flow chart would look like for troubleshooting modem problems at multiple sites (PDF Version):

The technique would be the same for different types of trouble calls. Strip the problem down to the smallest component, make sure it works (or doesn't work), then put stuff back one at a time until the problem returns. The last thing you connected when the problem returned is probably the culprit.

You can also do it backwards by disconnecting one thing at a time from the end until the problem clears, but I don't usually do that.

You need spares! If you don't have spares to check to see if some gizmo is bad, you're really going to bang your head against the wall. Anything electronic can break (and does). Replacing a device temporarily to see if the problem clears is often the only way to find the problem. One problem with having spares is when the previous tech using them didn't mark them as bad or questionable. Now you're going to bang your head against the wall anyway!

I personally keep a roll of gray tape with me for the bottoms of phones, and some orange tags for cards and phone line demarcs, to make note of my troubleshooting (both for myself and the next guy on the service call). I mark what I've done or what I suspect with a fine point Sharpie:

Almost every time I guess at a problem and skip proper troubleshooting, I waste more time than I could have gained. But there are a couple of cases I can think of that are so common that I'd start troubleshooting in the middle. Even then sometimes I get surprised and chase my tail.

The most common call I get where I assume the problem and the fix based on symptoms, without taking any measurements, is when a fire alarm panel throws a "communications failure" alarm.

In the old days companies would put dedicated lines on their fire alarm. The guys who designed the alarm panels didn't think companies would share these lines, that were supposed to save lives and property, with a phone system. Today, it seems like no company will put in dedicated phone lines for an alarm system, so I point callers to our Fire Alarm Tech Bulletin where the fix works 100% of the time.

The bottom line is that 99% of the time you can't troubleshoot or repair phone problems remotely... But you can do fewer service calls!

If equipment is installed so it can be unplugged and a spare can be plugged in by the end user, and the premise has spares on-hand or you can ship a spare there, you can save a lot of time and money. Your average customer may not go along with that, but if they're in a remote area or are otherwise hard to get a technician to, it may be a good way to handle service calls.

Handsets and cords are already modular on most phones and can be swapped with another phone to see if they are bad or intermittent, or replaced with a spare.

In the case of the missile bases they could make sure the wall phones are modular, show someone who works there how to swap a phone or other type of device with a spare, and just ship stuff up and back.

One of the problems with our missile bases is that the phones were likely installed before phones were modular. Even if installed later there may be special types of handsets or phones that aren't modular. In that case someone at the site should be able to unscrew the phone from the wall and splice wires with a spare with UY connectors (or just convert stuff to modular so the workers at the site can swap stuff out themselves).

In the old days the #1 service call an Interconnect phone man would go on is a dead CO line. And the problem was almost always with the phone company.

When I realized what it was costing us, I started making all lines modular before they went into the system on new installations with loop start lines and put a numbered cord from the system for each line into the numbered mod jack. I put a cheap ugly wall phone (so it doesn't get stolen) by the mod jacks so the customer could unplug the system, put the cord from the wall phone into the jack and test the lines themselves.

The modular cords were long enough so the customer could even swap a line above or below if there was an intermittent problem. So, if they had a problem on CO 3 they would swap the cords between the jacks marked CO 2 and CO 3 to see if the intermittent problem showed up on CO 2.

Unfortunately, my experience is that only 10% of the customers want to save the service call and troubleshoot CO line problems themselves. So, there are still a lot of service calls for lines that are dead from the phone company. Maybe you'll have better luck in your neck of the woods, or with specific customers?