Confessions of a Network Support Engineer

By Saba Jamaluddin

Many of you may have heard the urban legend of the customer who called tech support asking for the location of the “Any” key since the program on screen said “Press any key to continue”! However having spent time on both sides of the tech support call, I can safely say that “tech support” is a place where ANY thing can and will happen!

Network support is an odd business to be in. There are the odd customers who call in with simple non-issues while sometimes it is the support engineers themselves that are odd! (And I say this from experience having spent plenty of time with some up close! On recalling those days I can safely say it may have me who was the odd tech support engineer!) The following are simple scenarios that I am reminded of, from those days!

Once a ‘friend’ of mine who was helping a customer from South America was asked to assist via Instant Messenger. The customer, having given telnet access to the router, asked the engineer that they continue the conversation on an Instant Messenger session as the customer was not comfortable speaking in English and would rather type. Whilst looking at the router, the engineer discovered something that seemed a little out of place. All the while the IM window seemed to be scrolling with messages from the customer. To keep the customer from typing out more while the engineer looked into the problem, the support engineer typed out “Hold on please”. Straight forward right?!

Only to the engineer’s complete horror, what (s)he had typed out was actually reading “Hold me please” back on the screen! See how much trouble two little letters can be!!! Needless to say there was some embarrassment, some blushing, but plenty of laughs!

Of course there are some times when it is the customer who sounds infuriatingly incompetent and lacking in simple sense! Case in point, after nearly two hours of trying to see why VPN traffic would not go across the network, the customer was asked to check the integrity of the cable since there appeared to be no traffic entering the interface. To this the customer responded with, “There is no cable in the interface”. D-Oh!!!!

When asked why he had not shared this vital piece of information earlier so they could instead source traffic from the router itself rather than wait for it from the LAN, he responded “I didn’t know you wanted me to connect a cable”. Now one would usually treat the customer with respect assuming that they are fellow network professionals, but after two hours of mind numbing troubleshooting attempts, all you want to do is hit the mute button and let it all out!!

However, every now and then you will come across a case which baffles you for nearly two months only bring a great deal of amusement to you in the end. Allow me to explain. The customer would call in to say that they lose VPN connectivity intermittently. The cables were in tact. The traffic from the LAN enters the interface and is even seen leaving the WAN interface, yet it never would get to the far side. Keep in mind that this is a remote location with a WAN connection over a satellite link. The satellite dish resides atop a mobile van so when the people need to connect out in the field, they can.
Getting back to the case, according to the customer, the lengths for which the connection was broken would vary and these occurrences themselves were random. My colleague, having spent many years doing this, of course had tried all the tricks and then some.

The source of the problem came to light some two months later when the customer managed to get tech support on the phone at the same time as when the problem was on going and you will never guess how technical the nature of the problem actually was! It appeared that an even larger van would come and park next to this telecommunication van, blocking the line of sight connection, which then would cause the disruptions… intermittently!!

The moral of the story of course is as follows:

KISS: Keep It Simple Stupid! Not everything has to be due to some obscure, multi-lettered acronym. Sometimes the answers are very basic. Stick with the basics and work logically!

For those of us working in networking, rather than just change configurations at the drop of a hat and make everything about throwing more equipment at the problem, it may just work if you check the basics and follow the packet! If a packet enters the router, it should also leave the device through its “out” interface! The networking devices cannot make up or eat up packets. If you feed the device a packet, it is only natural that it should leave it too!

There are of course some obvious exceptions to this rule – when the router is actually eating up packets, but honestly that happens very seldom! That, if ever, should be the last check. It usually turns out to be a “precedence” issue, some route taking precedence so the packet chooses the “wrong” out interface. In the case of switches loops or broadcast storms can cause problems but then of course “know what you are doing!” when configuring!! Now servers – that’s an entirely different entity with a mind of its own and a life of its own, where whims and fancies defeat technical logic!

Just checking and double checking configuration is a good idea.

Two LANs are Better Than One!
To be honest, you cannot underestimate the power of collaboration. Learning from others’ experiences or simply discussing with your colleagues can lead to solutions. Some one somewhere may have run into your issue in the past and looking up friends and colleagues online in forums or down the hall, in cubes can lead to knowledge, conversation and coffee!! Discussions also helps start your own brain juices, describing a problem makes you think over things you may have overlooked.

(On too many occasions I have run out of a colleague’s cube mid sentence because in describing the problem out loud I had located the source of the problem!)
This I can state as being an important factor because too many folks for fear of appearing “inadequate” will not ask questions. If you don’t ask – how will you learn?

Networking is not part of our DNA (as yet) so none of us are born with this knowledge. Learning requires collaboration, mistakes, hands on work, some more mistakes and reading and experience, meaning repeat the cycle! Having said collaboration is important, collaboration should not imply you ride someone else’s coat tails, attractive as it may seem! Share your time and knowledge but please don’t expect that others will bail you out each time. Asking the same question once or even twice is okay but if you ask the same thing too many times you may be asked to RTFM. (Read the $%^ing Manual! No one said we were polite! )

Do not presume the customer is an idiot. Do not talk down to them. Most of the people calling network support are technology savvy and so speaking SLOWLY or being patronizing to them will only insult them.

Please remember the customer who pays your salary. You need them, please do not upset them! If they stop coming to your technical support group, what point is there to your work?

Listen: You cannot presume you know the customer’s issue. Give them the respect of listening to the issue. If you cannot ‘listen’ to the customer describe the problem, you will never be able to fix it.

Spend time giving yourself hands on training: Just reading about something or having your friend explain how it is done ‘does not an expert make’. You have to have experienced the frustration of configuring, testing and failure (did I say fail?) to truly understand what the customer feels. A baptism of fire so to speak! Like so many other fun things in life, there are no short cuts.

There are of course many issues, big and small that may be discussed when talking about networking, but honestly it is not rocket science. The devices themselves are not mysterious beings. They are dumb devices. What you configure on them, they will do, so, be sure you ‘get’ the concept before typing out commands. Prattling off commands by heart can only go so far, especially now with more and more devices with configuration input GUIs and automated ‘configurators’, what you need is to be sure you know where you want those packets to go. Know who, between you and the box, is the more intelligent one.

Work out the logic. The configuration shall follow.

Saba is a network security specialist based out of Pakistan. She can be reached at saba.jamaluddin@gmail.com