[gPXE] any negative implications of setting TCP window size to (65536 - 4) ?

[Marty Connor]

Joshua Oreman wrote on 2/1/10 6:38 PM:
> On Mon, Feb 1, 2010 at 6:08 PM, Peter Scheie<peter at scheie.homedns.org>  wrote:
>> The default setting for the TCP window size, as set by #define
>> TCP_MAX_WINDOW_SIZE in src/include/gpxe/tcp.h, is 4096. With this setting, it
>> was taking 8-10 minutes for gpxe to download my 22MB initramfs over the
>> internet.  At the suggestion of people on #etherboot, I changed it to 65536 - 4,
>> which was listed in tcp.h but commented out in favor of the 4k setting. At this
>> new setting, downloading the 22MB file takes only about 30 seconds!
>>
>> I understand that gpxe has no ability to handle out-of-order packets, so a
>> possible negative effect is that if there is disruption in the packet flow, the
>> whole window will need to be resent.  But in my testing so far, even if that is
>> happening, downloading is still much faster.  Are there any other potential
>> pitfalls from using the larger window size?  Why is the default set so low?
>
> I think the problem relates to gPXE's memory constraints. For various
> reasons relating to boot-time memory constraints, we use a fixed-size
> heap; currently its size is set at 128kB, and a 64kB window size would
> take fully half of that space away. Some network drivers need lots of
> memory too (sky2 for example).

This seems like kind of a tricky constant to set.  The core doesn't know 
what kind of buffer space for rx the driver has allocated, but maybe the 
driver could tell it, and it could try to set the window size to hold 
(at least) the number of packets the network card can accept.

I think with 4096 (4KB) as a window size, and the size of segments we 
are receiving, we could only hold 2 * 1536byte packets before the window 
is full.

> On the other hand, we're probably safe bumping it to 256kB. Thoughts
> from other developers?
> -- Josh

I think 256kB must be a typo because that's twice the heap size.  Did 
you mean 16KB or 24KB?

I was going to run some WAN tests to see what the difference in download 
speed was with various sizes, like 4KB, 8KB, 16KB, and 32KB.
Peter, have you tried any other settings?  It would be nice to get more 
data points.

I have a suspicion (didn't look) that each connection is going to 
allocate this amount of memory, so with a lot of connections, things 
could get ugly real fast.  Put 3 NICs in a machine with those settings, 
and let's see what happens when they all meed to do DHCP :)

With the amount of performance improvement that Peter has reported, it 
looks like there might be some excellent opportunities for improvement 
of our TCP performance such as:

  - Out of order packet handling.
  - Dynamic window sizing.

That's my off-the-top-of-my-head-just-before-I-go-to-bed take on this.

( perhaps some enterprising prospective GSoC student would like to look 
into this :) )

/ Marty /