TCP and mobile IP

Friday, September 4. 2009

Posted by Marc 'Zugschlus' Haber in Computer und Netze at 16:00 | Comments (15) | Trackbacks (0)

Steinar H. Gunderson, sesse, has written an interesting article about TCP performance. I didn’t find your blog’s comment function, so I am commenting with a trackback. (note: which didn’t work either, “The auto-discovered trackback URI does not match our target URI”)

I frequently use mobile internet, using various of the German GSM/UMTS network operators, out of a moving train. As you have written, this frequently causes packet loss which is not only not caused by congestion, but sends the congestion avoidance algorithms on a false path.

For example, when the train passes through the 3575 m long Distelrasentunnel between Frankfurt and Fulda, my network link is broken for like two minutes. Passing through other parts of Germany sometimes gives me a ping response of hundreds of thousands of microseconds by virtue of the rather huge send buffer the UMTS equipment has.

In these circumstances, ssh sessions frequently take tens of minutes to notice that the network is back before the session is useable again. Frequently, it doesn’t come up again before an hour has passed. And I have not found a way to work myself around this. Can you explain what’s happening here, and do you have any ideas to solve the issue?

Defined tags for this entry: debian-english, english, network, tcp, umts

Random Entry: Automatic initialization of a Option 3G Datacard
< grml als eigenes Rescuesystem | Booting from a large hard disk II >

Mark as read

Trackbacks

Trackback specific URI for this entry

No Trackbacks

Comments

Display comments as (Linear | Threaded)

Good question; I don’t have an answer for you, other than to suggest running screen, tmux, or the like on the remote host and reconnecting as necessary. (To be sure, that’s not the greatest solution because resuming login sessions can interact poorly with X11 or agent forwarding, but it still beats losing them altogether, and supplies other useful features as well.)

In general, I’ve been under the impression that the internet has grown less friendly to long-lived TCP sessions over the years, with the bulk of users just caring about HTTP. :-/

Comment (1)

#1 Aaron M. Ucko (Homepage) on 2009-09-04 20:22 (Reply)

yes, of course, screen is the obvious answer, but that makes some of my processes unnecessarily clumsy, and I lose konsole’s excellent scrollback capabilities since screen interferes. I’d rather have the network stack handle the situation gracefully.

Comments (6)

#1.1 Marc 'Zugschlus' Haber (Homepage) on 2009-09-04 21:00 (Reply)

I put this into my ~/.screenrc in order to force screen to let my terminals maintain scrollback the way I’m used to:

# gimme back my scrollback in xterm!
termcapinfo xterm|xterms|xs ti=\E7\E[?47l

What it does is change the terminal init sequence and removes the “switch to the alternate (scrollback-less) screen buffer” bit that’s in the standard terminfo entry for xterm.

And here’s the rest of my .screenrc; without this I find screen too obnoxious to be usable:

startup_message off

# I can’t live without my ^a in bash/readline
defescape ^bb
escape ^bb

# probably don’t need this any more, since I use the native xterm scrollback
defscrollback 1000

Comment (1)

#1.1.1 Marius Gedminas (Homepage) on 2009-09-05 00:53 (Reply)

That’s nice, I’ll use it in the future. It’s however a pity that TCP cannot be tuned.

Comments (6)

#1.1.1.1 Marc 'Zugschlus' Haber (Homepage) on 2009-09-06 15:09 (Reply)

This isn’t exactly an answer to your question, but it might help in case your usage is mostly wireless based: switching the default Linux congestion management algorithm to TCP Veno. It’s theoretically optimized for wireless links and should interpret lost packets in a way less damaging to throughput.

I haven’t tested this specific algorithm, but I did play a bit with other high-throughput ones and the difference in performance was sensible, though not particularly shocking.

This article is a nice overview of the different algorithms available:
http://linuxgazette.net/135/pfeiffer.html

(you might have to recompile your kernel, my standard Debian kernel is showing only cubic and reno as available algorithms...)

Comments (2)

#2 Leo 'costela' Antunes (Homepage) on 2009-09-04 21:00 (Reply)

That is important information, rebuilding my kernel to try.

Comments (6)

#2.1 Marc 'Zugschlus' Haber (Homepage) on 2009-09-04 21:14 (Reply)

oops, I take it back, the right algorithms are there as modules, they just weren’t loaded... so: no need to recompile!

Comments (2)

#2.2 Leo 'costela' Antunes (Homepage) on 2009-09-04 21:17 (Reply)

ConnectTimeout 30
ServerAliveInterval 30

try adding that to ssh conf

Comment (1)

#3 XANi on 2009-09-04 21:04 (Reply)

I have that in my config for years. Doesn’t help in these situations. Actually, I think it makes things worse since idle sessions might not even not notice the outage without ServerAliveInterval.

Comments (6)

#3.1 Marc 'Zugschlus' Haber (Homepage) on 2009-09-04 21:12 (Reply)

The basic property of standard TCP is that it thinks packet loss is caused by congestion, whereas in wireless environments that is rarely so, instead it’s caused by actual data loss, meaning these two are pretty much incompatible from the get-go. TCP sees loss and then it starts trying to resend data with reduced speed, which ends up causing more and more packets getting lost, making it think that you’re on a very slow link, rather than just an unstable relatively fast link (which is what modern 3G networks are like). This is a documented fact, or so I was told at my university classes a year ago (wow, actual real-world use for those...).

My actual experience with ssh sessions on a train (Zagreb-Vinkovci, FWIW) was based on those factoids and on common sense :) I was usually able to learn quickly that I was going to enter a small period of downtime, seeing a part of the route where a BSS handover was likely - outside of populated areas etc - so I was forced to learn that, rather than trying to improve it, it’s best avoid TCP congestion avoidance altogether. So I do two things:

First, use ssh only for processes where I trigger output changes, e.g. mutt. That way you retain control over when and how much new data comes (or tries to come) over the link. For example if you’re running any form of instant messaging (like ssh session with irssi in it), you can’t control how much new data will try to be pushed to your screen or when it will happen, so it’s inherently incompatible with semi-random blackout periods.

Second, go idle on the active ssh connection as soon as possible when, best if I don’t send a single byte over a dead link. As soon as the link comes back up, start up mtr (or have a mtr running all the time if it won’t eat up your quota) and wait for proof that actual IP connectivity has come back up. Then resume the SSH session.

Hope that helps.

Comment (1)

#4 Josip Rodin on 2009-09-05 00:37 (Reply)

It is strangely disturbing that the TCP definitions and implementations weren’t changed to cater for changed environments. And I absoutely hate the idea of having to adapt my work methods depending on what kind of network link I am on.

Comments (6)

#4.1 Marc 'Zugschlus' Haber (Homepage) on 2009-09-06 15:11 (Reply)

A coworker had told me, that setting net.ipv4.tcp_low_latency
on both sides would help. I’ve no idea what this option does,
and Documentation/networking/ip-sysctl.txt is quite unhelpful.

Comment (1)

#5 Maurice Massar on 2009-09-09 13:42 (Reply)

tcp_low_latency shouldn’t have any effect on this problem, since AFAIK it doesn’t have any effect on the retransmission timeout (RTO), which is the main factor causing those long recovery times.

What you really want to change is TCP_RTO_MAX, which is 120 seconds by default on linux. This forces the stack to do retransmits earlier, so connections will recover faster once connectivity is restored. Unfortunately, patches adding a sysctl to change it at runtime have repeatedly been dropped, so you have to patch include/net/tcp.h and recompile the kernel.

Additionally, you might want to increase net.ipv4.tcp_retries2 (the maximum retry count for established connections) to avoid the kernel breaking connections earlier due to shorter RTO. Note that this only affects the sending direction, so the kernel has to be modified on both ends of the ssh connection.

A negative side effect is that the TCP stack behaves worse if a connection really has a very high latency (more spurious retransmits). It might be unwise to reduce the max RTO to much on a public server. The RFCs mention 60 seconds as a lower limit.

A completely different solution might be using rocks (http://freshmeat.net/projects/rocks). However, this project isn’t maintained for quite some time now, and I don’t know if it is still usable today.

Comment (1)

#6 Andreas Ferber on 2009-09-29 00:47 (Reply)

That’s really insightful. Even if it doesn’t offer a viable solution for me, I have learned a lot. Thanks!

Comments (6)

#6.1 Marc 'Zugschlus' Haber on 2009-09-29 09:25 (Reply)

No Roks isn’t useable anymore it carshes all the time, it is ment to handle a different problem though. Rocks is to be able to keep a TCP connection even though you change IP..

Comment (1)

#6.2 Erik J on 2009-10-15 14:45 (Reply)

Add Comment

Name
Email
Homepage
In reply to
Comment	HTML-Tags will be converted to Entities. To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly. Enter the string from the spam-prevention image above: Enclosing asterisks marks text as bold (word), underscore are made via _word_. Gravatar/Favatar supported
	Remember Information?

TCP and mobile IP

Zugschlusbeobachtungen

Friday, September 4. 2009

TCP and mobile IP

Quicksearch

Blog Administration

Categories

Recently added

Comments

Syndicate This Blog

Show tagged entries

Template dropdown

Technorati