[gPXE] [Fwd: Re: any negative implications of setting TCP window size to (65536 - 4) ?]

[Peter Scheie]

Marty Connor wrote:
> Joshua Oreman wrote on 2/1/10 6:38 PM:
>> On Mon, Feb 1, 2010 at 6:08 PM, Peter 
>> Scheie<peter at scheie.homedns.org>  wrote:
>>> The default setting for the TCP window size, as set by #define
>>> TCP_MAX_WINDOW_SIZE in src/include/gpxe/tcp.h, is 4096. With this 
>>> setting, it
>>> was taking 8-10 minutes for gpxe to download my 22MB initramfs over the
>>> internet.  At the suggestion of people on #etherboot, I changed it to 
>>> 65536 - 4,
>>> which was listed in tcp.h but commented out in favor of the 4k 
>>> setting. At this
>>> new setting, downloading the 22MB file takes only about 30 seconds!
>>>
>>> I understand that gpxe has no ability to handle out-of-order packets, 
>>> so a
>>> possible negative effect is that if there is disruption in the packet 
>>> flow, the
>>> whole window will need to be resent.  But in my testing so far, even 
>>> if that is
>>> happening, downloading is still much faster.  Are there any other 
>>> potential
>>> pitfalls from using the larger window size?  Why is the default set 
>>> so low?
>>
>> I think the problem relates to gPXE's memory constraints. For various
>> reasons relating to boot-time memory constraints, we use a fixed-size
>> heap; currently its size is set at 128kB, and a 64kB window size would
>> take fully half of that space away. Some network drivers need lots of
>> memory too (sky2 for example).
> 
> This seems like kind of a tricky constant to set.  The core doesn't know 
> what kind of buffer space for rx the driver has allocated, but maybe the 
> driver could tell it, and it could try to set the window size to hold 
> (at least) the number of packets the network card can accept.
> 
> I think with 4096 (4KB) as a window size, and the size of segments we 
> are receiving, we could only hold 2 * 1536byte packets before the window 
> is full.
> 
>> On the other hand, we're probably safe bumping it to 256kB. Thoughts
>> from other developers?
>> -- Josh
> 
> I think 256kB must be a typo because that's twice the heap size.  Did 
> you mean 16KB or 24KB?
> 
> I was going to run some WAN tests to see what the difference in download 
> speed was with various sizes, like 4KB, 8KB, 16KB, and 32KB.
> Peter, have you tried any other settings?  It would be nice to get more 
> data points.
> 
I did try one other setting, 48K (or was it 40K?), and then the time to download
my initramfs increased from ~30 seconds to ~2 minutes, although my recollection
is a bit fuzzy, and I may have had some other networking strangeness going on at
the time so that latter figure is not very reliable.  For my upcoming demo, I
want the most impressive time, so I went back to (65536 - 4) for the time being.
  I will try a variety of memory settings and post my results to the list.

Peter