RoadRunner - a stupid story... long too.

[Blinky]

This long and boring unless you're into stupid support stories. Most of it you can gloss over anyway because you get the idea pretty quick that no one ay Time-Warner in Raliegh KNOWS HOW TO SIMPLY THINK FOR THEMSELVES! I don't suggest reading it in detail, I'm writing it for cathartic reasons... but you should read the conclusion at the end.

On wednesday or thursday I started noticing that I was getting more than the usual, every-once-in-a-while, timeouts in Firefox. I also noticed that it would take me a few tries to get my email and then realized that people had been calling on my Vonage phone and it wasn't even ringing. I started poking around and checked the network status for RoadRunner, my ISP. I would show you a screen shot of the 30 odd messages indicating they have been experiencing unexpected internet connectivity problems... but that site is offline now so I can't grab a screen shot.

So, I waited, thinking they'd get it fixed... and I waited until last night (Saturday) when in a fit of aggravation I contacted them through their little 'live chat' help desk... which forced me to use IE and install some fairly invasive ASP code so they could 'support' my computer... assholes.
Anyway, the girl gets on the chat thing and asks me a million and one questions, many of which I had already answered just to start the chat thing up. Then I explain the problem and she declares that no disconnects have occurred... I tried to tell her that no disconnects occured when I was stating my problem but she must've missed it. Long story short she made it clear their was nothing wrong on their end and that the network status thing just hadn't been updated recently... the last message was nearly 12:00 midnight on 1 September. Great, screw you. She never answered my question about DNS problems which I've been suspecting the whole time.

So this am I send the RR email support a message and heard nothing back for a few hours so I sent them another one. This is it...

This is the second email message I've sent regarding this issue. I have not received a confirmation or been contacted regarding remedy.

I am experiencing intermittent but very frequent timeouts connecting to internet resources. The behavior started on Wednesday or Thursday and renders my internet service practically unusable.

This is a new behavior after a long period of acceptable performance. Nothing on my system has been changed recently. I'm an advanced user and have checked my system thoroughly for configuration problems and mal-ware.

This is a screen shot of the browser connection timeout error, note that this is the North Carolina Road Runner website...
http://www.chiphildreth.com/_Images/ncrr.jpg

I'm also getting 'unable to connect to server' errors.

When I ping the RR DNS (dns4.rr.com) packets drop every time, usually at least two, sometimes all four... all four haven't made the trip since I've been testing. This is a textfile log of a typical ping test:


Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\chip>ipconfig /all

Windows IP Configuration

Host Name . . . . . . . . . . . . : peewee
Primary Dns Suffix . . . . . . . :
Node Type . . . . . . . . . . . . : Mixed
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : nc.rr.com

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . : nc.rr.com
Description . . . . . . . . . . . : 3Com 3C920 Integrated Fast Ethernet
Controller (3C905C-TX Compatible)
Physical Address. . . . . . . . . : 00-06-5B-DC-85-59
Dhcp Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IP Address. . . . . . . . . . . . : 192.168.0.2
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.0.1
DHCP Server . . . . . . . . . . . : 192.168.0.1
DNS Servers . . . . . . . . . . . : 192.168.0.1
Lease Obtained. . . . . . . . . . : Saturday, September 02, 2006 11:25:4
3 PM
Lease Expires . . . . . . . . . . : Tuesday, September 05, 2006 11:25:43
PM

C:\Documents and Settings\chip>ping dns4.rr.com

Pinging dns4.rr.com [65.24.0.172] with 32 bytes of data:

Reply from 65.24.0.172: bytes=32 time=56ms TTL=241
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 65.24.0.172:
Packets: Sent = 4, Received = 1, Lost = 3 (75% loss),
Approximate round trip times in milli-seconds:
Minimum = 56ms, Maximum = 56ms, Average = 56ms

C:\Documents and Settings\chip>ping 65.24.0.172

Pinging 65.24.0.172 with 32 bytes of data:

Request timed out.
Request timed out.
Request timed out.
Request timed out.

C:\Documents and Settings\chip>ping dns4.rr.com

Pinging dns4.rr.com [65.24.0.172] with 32 bytes of data:

Reply from 65.24.0.172: bytes=32 time=54ms TTL=241
Request timed out.
Request timed out.
Reply from 65.24.0.172: bytes=32 time=55ms TTL=241

Ping statistics for 65.24.0.172:
Packets: Sent = 4, Received = 2, Lost = 2 (50% loss),
Approximate round trip times in milli-seconds:
Minimum = 54ms, Maximum = 55ms, Average = 54ms

C:\Documents and Settings\chip>ping 65.24.0.172

Pinging 65.24.0.172 with 32 bytes of data:

Reply from 65.24.0.172: bytes=32 time=55ms TTL=241
Request timed out.
Reply from 65.24.0.172: bytes=32 time=54ms TTL=241
Request timed out.

Ping statistics for 65.24.0.172:
Packets: Sent = 4, Received = 2, Lost = 2 (50% loss),
Approximate round trip times in milli-seconds:
Minimum = 54ms, Maximum = 55ms, Average = 54ms

C:\Documents and Settings\chip>ping 65.24.0.172

Pinging 65.24.0.172 with 32 bytes of data:

Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 65.24.0.172:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

C:\Documents and Settings\chip>ping 65.24.0.172

Pinging 65.24.0.172 with 32 bytes of data:

Reply from 65.24.0.172: bytes=32 time=55ms TTL=241
Reply from 65.24.0.172: bytes=32 time=55ms TTL=241
Request timed out.
Request timed out.


This leads me to believe the problem is on the RR side of the demarc.

I've already discussed this with a RR tech via live chat who stated there were no disconnects in the last two days and simply refused to investigate the matter. When I asked about DNS she didn't answer. The issue is not one of disconnects, it's a performance issue and seems likely to stem from failed DNS lookups.

Hopefully this is enough information to get past the usual 'disconnect your modem and wait 3 minutes' routine. I would like someone with working knowledge about IP connectivity and DNS to investigate the problem.

Please acknowledge the receipt of this support request.

Thank you,
Chip

They responded with an email asking some questions, this is that email with my answers:

W-Raleigh wrote:
> Dear Chip,
>
> Thank you for writing and thank you for choosing Time Warner - Road
> Runner E-mail Technical Support.
>
> I understand that you have an intermittent connection.
>
> I would be glad to help with this.
>
> Intermittent connectivity to the Internet is the result of one or more
> of the factors described below. Please follow the suggested guidelines
> and respond with the answers to every question so our specialists have
> the information they need to determine the cause of the intermittence
> you're experiencing.
>
>
> PREVENT OVERHEATING
>
> It is important to ensure the cable modem is not overheating. Do not
> place it on a carpeted surface. Ideally, the modem should be in a cool,
> dry place. Always make sure there is plenty of space around it,
> particularly the side with the ventilation slots. Keep the modem several
> feet away from areas of high electromagnetic interference, this includes
> televisions, computers, monitors, speakers, and microwaves.
>
My modem is standing vertically on top of a desk with lots of air space surrounding it. The modem is slightly warm to the touch.
> RESET CABLE MODEM
>
> To ensure the information the modem is providing to your computer is
> accurate, its memory must be reset. If you have a router connected to
> your cable modem, you must disconnect the router to verify it is not the
> cause of or a contributor to the problem. After disconnecting the router
> and unplugging its power, the modem must be reset. Do this by unplugging
> the modem's power cable from the outlet for two minutes; when the power
> is plugged back in and the "Cable" or "Online" light stops flashing and
> remains on, turn on or restart the computer. This process synchronizes
> the information between our network, the cable modem, and your computer.
>
> If you have a router and bypass it, you should monitor the connection
> over the next few days. When connectivity is lost, note which lights on
> the modem are on, off, or flashing and whether restarting the computer
> alone solves the problem or if the modem needs to be unplugged for a
> couple minutes before restarting the computer. This will help us
> diagnose where the problem may lie.
>
I have reset the modem and router twice, once waiting about 10 minutes before re-applying power and a second time waiting about 2 hours.
I'm not willing to expose my computer to the internet without a NAT device and cannot do those tests. But the modem is not dropping offline, the problems are related to internet response. I can click a link twice in a row and get timeouts then click a third time and get a normal response.
> SOFTWARE
>
> Software problems such as misconfigured network or router setups,
> problematic network card drivers, and interference from firewall
> software can hinder the connection. Many software firewall programs
> include a lock feature which automatically disables the Internet
> connection when no activity has occurred for awhile. The EZ Armor
> Firewall software has this feature. To disable it, click on the Program
> Control tab and set the Automatic Lock to 'off'. For other firewall
> software, please refer to the help menu, included documentation, or
> contact the manufacturer of the program for further assistance.
>
> If the computer has been left idle, it may have entered a low power
> standby, suspend, or hibernation mode. Such power management features
> often disable the network card and USB interfaces; these may not be
> properly re-enabled when the computer comes out of this mode. Restarting
> the computer will force the re-initialization of all hardware and
> software. For information or assistance with disabling power management
> settings, please contact the vendor you purchased the computer from, the
> computer manufacturer, or a PC technician.
>
The configurations of my router and computers have not changed in a long time... at least not to my knowledge. I've checked them and everything appears to be normal. I don't use a software firewall for security, I rely on NAT, manual port management and mal-ware scanning. After installing Service Pack 2 for WinXP I found that it enabled the Windows Software Firewall but I disabled it... that was about a month ago.
> HARDWARE
>
> Malfunctioning hardware (such as network cards, routers, and
> hubs/switches) can make it difficult to obtain or maintain an Internet
> connection. Problems with the network card in the computer or with
> networking devices attached to the modem will affect the connection in
> unpredictable ways. This is why networking devices like routers and hubs
> must be disconnected while troubleshooting.
>
I'm using 3 machines right now and all are exhibiting the timeout behavior so I don't suspect client side hardware unless I have a jabbering card. Data transfers between computers appears to be normal however. I'll investigate the noisy card possibility. I'll also look into the Ethernet switch... I have a spare one I can use to test with.
> INFRASTRUCTURE
>
> Low quality cable lines within the house or between the house and the
> local office can result in signal strength fluctuations that cause the
> connection to drop. This situation can only be addressed by the
> specialists at the local office after all other possibilities have been
> explored and you have provided the information and answers requested
> below.
>
I don't have reason to suspect the lines unless they suddenly degraded... which seems a remote possibility.
> If you've abided by the above recommendations and continue to experience
> the intermittent loss of connectivity, the answers to the questions
> below will help us determine how to proceed.
>
>
> Have you verified that all the physical connections to and from the
> modem are secure? YES
>
> What brand and model of cable modem do you have (the information is
> written on the modem itself)? Motorola SB4220
>
> Is the computer connected to the cable modem with an Ethernet or USB
> cable? Ethernet
>
> What is the status of each of the cable modem lights when you encounter
> a connection problem? Top four lights are steady green, Activity light is flashing amber (no pattern), standby light is off.
>
> When did this connection problem begin? I think Wednesday or Thursday. I'm not certain because I usually suspect a busy website first when I get a timeout. It took me a while to decide it was on my end.
>
> How often do you lose the connection? As I said, I'm not actually losing the connection, but I get timeout or server not found error more than 50% of the time now. It will probably take me more than one attempt to send this message. Very frustrating.
>
> Is there any pattern to when the connection drops? Is it affected by
> time of day, does it occur after a long idle period, or is it random? None that I can discern, the problem has been steady except for a couple of brief periods that lasted no more than an hour.
>
> To regain connectivity do you have to reboot the modem, or restart the
> computer, or does it come back on it's own? Nope, I just retry the link or the send button.
>
> Do you have cable TV? If so, are there any reception problems like snow,
> ghosting, or pixelization? No cable TV.
>
> Are you running any sort of home network or router? If so, have they
> been bypassed for testing? Like I said before, I'm not willing to risk it though I will test the switch by using a different one.
>
> What is the IP address the modem has assigned to your computer? What are
> the IP Lease Obtained and Lease Expiration dates and times? Currently the WAN IP is xx.xx.xx.xxx and appears to be persistant (at least while offline for 2 hours); I don't have a tool to determine the expiration. Inside I'm using DHCP on two machines and an assigned IP on one (all on the public address space). DHCP leases are on a 3 day expiry.




This was their response:

Dear Chip,

Thank you for your response. As routers are one of the top reasons for
problems accessing the internet, we require that the router be bypassed
for troubleshooting. At this time, there are no known outages in your
area, and the local technicians will not be able to resolve the issue
without the troubleshooting that is required.


If you have any further questions or problems, please feel free to
e-mail us again for further assistance. When replying to this address,
please include this message as well as all previous correspondence
regarding this issue.

Thank you for choosing Time Warner - Road Runner and have a great day!
Time Warner - Road Runner Technical Support Department.

Coming from the desk of Technical Support Agent Jeremy, 15151


************************************************** *********

In an ongoing effort to continue improving our quality of service, we
are conducting a customer survey. If you would like to participate,
please click on the following link: http://csat.rr.com/emailsurvey.html

That's very nice... I'll be filling out a customer survey alright.

The bottom line is, I took a nap. When I got up about 1-1/2 hours later I pinged the DNS and got all 4 packets, round trip under 30ms! I did it again, same thing. I did it again and again and again... no lost packets all trips were <30ms. I get onto the Camel and things are snappy as hell, thay have been now for quite a while. IMAGINE THAT!

Oh no Mr. Hildreth, you have to expose your system to the net without any protection so we can troubleshoot... your router must be the problem.
Funny, I haven't changed a thing, I haven't even re-booted and miraculously everything is better... after four days of less than dial-up performance.
You draw your own conclusions.

Chip

[Mr. Pickles]

Bottom line is "you took a nap"?

I think I will now in about 1 min. Thanks, and good luck to ya.....

[brian.austin]

Why would DNS issues cause "dial up performance"? After the intial lookups, there should be no changes until the DNS Time To Live (TTL) setting for your workstation's cache. In other words, once you got the name to IP mapping, you shouldn't see a speed issue until you have to visit another site.

Banner ads can cause a slowdown if they are being fed from different users.

Easy way to see if it's truly a DNS issue is to open up three or four DOS windows and ping -t three different servers, including the possible bad DNS server. Yahoo will allow pings and several UUNet servers will answer them (I use their DNS caching servers at 198.6.1.1-5). Ping all of them at the same time, using the -t switch to do a continuous ping. Then try to browse the Internet.

If all of the servers drop connections during the pings, you've got connection issues (which could be further upstream from you). If only the DNS server drops, it's their DNS and nothing else. By doing the pings at the same time, you eliminate network variances as much as possible.

Incidentally, Saturday nights are usually router firmware/config upgrades and maintenance periods. If anything suddenly goes away overnight on Saturdays, it's usually this that fixed it.

[Blinky]

Brian,
Looking at sites like the Camel triggers a lot of DNS lookups because the image files are hosted all over the place; same with lots of websites. Page content nowadays is often on multiple hosts. I was able to ping my web server by IP reliably... it was their DNS that was dropping packets.

Frankly, I'm really burned out on trying to convince people this was a problem on RR's side of the demarc. I really don't give a shit if it was the DNS or one of their routers or what... I simply wanted someone with some skills to actually take a look at it... which apparently someone did... about 3:00PM on Saturday. The RR net status showed no scheduled maintenance for the last few weeks... but it did show numerous unexpected outages... nobody at RR support ever admitted they had an outage even though their own website was eaten up with 'red' messages. Nobody at RR ever acknowledged they even knew what a DNS was, much less that they checked it out.

The problem resolved within 2 hours of me rattling cages... after four days of serious performance problems... it wasn't maintenance. I don't give a damn what it was but having spent a lot of time checking out my end and then looking at what little of their end I could look at, I think their DNS was spotty.

What gripes me is that they never copped to having a performance degradation, from maintenance or anything else. The minute FTTP is available to me... RR is history.

[brian.austin]

Originally Posted by B

Frankly, I'm really burned out on trying to convince people this was a problem on RR's side of the demarc. I really don't give a shit if it was the DNS or one of their routers or what... I simply wanted someone with some skills to actually take a look at it... which apparently someone did... about 3:00PM on Saturday. The RR net status showed no scheduled maintenance for the last few weeks... but it did show numerous unexpected outages... nobody at RR support ever admitted they had an outage even though their own website was eaten up with 'red' messages. Nobody at RR ever acknowledged they even knew what a DNS was, much less that they checked it out.

Easy, big guy. I was only trying to give you more ammo if you were still firing.

The last thing ANY ISP will ever admit to is unscheduled downtime. Service Level Agreements (SLAs) mean lost revenue for those paying attention to them. Admitting unscheduled downtime means customer's with a firm grasp of their SLAs get credits on their bills, even for performance degradation. Hate to say it but it's not unusual. The more you know and measure independently, the more ammo you have for these types of issues.

Next time, ask for the ticket # and escalate if it's not satisfied. Escalation is the key to getting something fixed that Tier 1 and 2 tech support can't seem to resolve. Every Help Desk is designed to resolve an issue with the first call (called FIrst Call Resolution in our world). FCRs are good. Non-FCRs are bad. Escalations are worse, since they mean not only could the issue not be resolved but the customer is major pissed off about it. Theoretically, a ticket can be escalated all the way to a VP somewhere. The highest I've ever gotten personally was a regional supervisor. At that point, it's mainly a manager yelling at some techs to get their ass moving. (I've been the manager, too)

Sorry for your troubles. Personally, I have several ISPs to choose from in a variety of environments. If my Cox connection isn't working well, I've got Sprint EVDO on my cell phone that connects to either computer. If that isn't working, my neighbor's AP is wide open and I can use his Qwest DSL line. If they're all down, I've probably got something better to do anyway.

[timjon]

It could have been a dodgy router somewhere in their network, which would result in dropped packets to everywhere, including their DNS server. Did you try running pings to other servers, or running traceroutes to determine exactly which hop was dropping packets?

I agree their tech support sounds laughably clueless, though. In future, if you KNOW that the problem is in their network, you could just say "yes, I've connected directly, and still have the same problem". If you tell them you haven't done something on their pointless little checklist, even if it's clearly irrelevant, they'll seize on it as an excuse to ignore you.

Cheers,

Tim.