[gPXE] any negative implications of setting TCP window size to (65536 - 4) ?

[Glenn Brown]

> The lack of out-of-order reception [...]  really needs to be addressed.

To put some numbers on the performance impact of out-of-order packets: a 
1997 paper reported "Overall, 2.0% of all of [...] data packets and 0.6% 
of the acks arrived out of order".  That's from
http://cseweb.ucsd.edu/classes/wi01/cse222/papers/paxson-e2e-packets-sigcomm97.pdf
section 3.  I used this paper only because it was the first source I 
found via Google.

Those statistics are enough to estimate the effect of dropping instead 
of reordering: in-order data packets will come in bursts of 50 packets 
on average and have a ~round-trip-time delay, say 200ms, before the next 
burst.  With 1500 byte MTU on an 'infinite bandwidth' network, that 
limits performance to ~3Mbit/s.

This thread was started because gPXE's limited window size was 
throttling performance.  For comparison, gPXE's 4K window and the same 
assumed 200ms round trip time limits performance to ~4K/200ms or 
0.163Mbit/s.  So, it seems the window-size problem is a much more severe 
performance limitation than is the lack of out-of-order support.

The window size must be addressed before out-of-order support will help.

That said, out-of-order is not hard to address.  It simply requires a 
sorted list of 'early' packets per connection, which is checked after 
each in-order packet is processed.  The only complexities that spring to 
mind are "What's the right way to deal with buffer starvation?" and "Can 
one avoid copies when buffering early receives?"  I'd be happy to share 
thoughts on these if anyone is planning to implement it.  Unfortunately, 
I don't have time to implement+debug+support it myself.

--Glenn