TCP socket buffer size and iSCSI

[Padmanabhan]

Hello All,
For a software iSCSI Initiator/Target over Gigabit ethernet switched network, how TCP socket sizes affect it's performance ?
Do iSCSI Initiator/target set TCP buffer size or uses default values ?
 
Most of the times data transfer will be unidirectional over a TCP connection either storing it or retrieving the data . Is there any guidelines for TCP buffers for iSCSI ?
Any pointers to docs on this topic would be great

Thanks


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Iscsitarget-devel mailing list
Iscsitarget-devel@...
https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel

[rswwalker]

Padmanabhan wrote:

> Hello All,
> For a software iSCSI Initiator/Target over Gigabit ethernet
> switched network, how TCP socket sizes affect it's performance ?
> Do iSCSI Initiator/target set TCP buffer size or uses default
> values ?
>  
> Most of the times data transfer will be unidirectional over a
> TCP connection either storing it or retrieving the data . Is
> there any guidelines for TCP buffers for iSCSI ?
> Any pointers to docs on this topic would be great

In the past IET use to set the window sizes upon start-up, but
it has been determined that the built-in capabilities of the
TCP/IP stack with the 1323 options enabled do a much better
job at auto-negotiating these during operation.

I advise making sure both targets and initiators have the TCP
1323 options enabled and let them negotiate the send/receive
window sizes. Only if you encounter performance issues due
to buggy implementation of the 1323 options should you manually
configure them.

I think most places recommend a 1MB send/receive window size
if you MUST manually configure it. If left to auto-negotiate
these windows can scale up to 8MB or more on large block
io transfers.

-Ross

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Iscsitarget-devel mailing list
Iscsitarget-devel@...
https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel

[Padmanabhan]

I did a capture at the initiator side and the window sizes announced by Initiator/Target  are around 57XX/58XX bytes...

From RFC 1323 Sec 2.1

""The maximum receive window, and therefore the scale factor, is determined by the maximum receive buffer space. In a typical
modern implementation, this maximum buffer space is set by default but can be overridden by a user program before a TCP connection is
opened.""

From man pages of TCP and output of sysctl

net.ipv4.tcp_mem = 194112  258816  388224
net.ipv4.tcp_wmem = 4096   16384    4194304
net.ipv4.tcp_rmem = 4096     87380    4194304

tcp_rmem        
     This  is  a  vector of 3 integers: [min, default, max]. These parameters are used by TCP to regulate receive buffer sizes.
     TCP dynamically adjusts the size of the receive buffer from the defaults listed below, in  the  range  of  these  sysctl
     variables, depending on memory available in the system.
    
     min
      - minimum size of the receive buffer used by each TCP socket.The default value is 4K, and is lowered to PAGE_SIZE bytes
        in  low-memory systems.  This value is used to ensure that in memory pressure mode, allocations below this size will
        still  succeed. This is not used to bound the size of the receive buffer declared using SO_RCVBUF on a socket.
 
    default 
      -  the default size of the receive buffer for a TCP socket.  This value overwrites the initial default buffer size from
         the generic global net.core.rmem_default defined for all protocols.  The default value is 87380 bytes, and is lowered
         to  43689 in  low-memory systems.  If larger receive buffer sizes are desired, this value should be increased
         (to affect all sockets). To employ large TCP windows, the net.ipv4.tcp_window_scaling must be enabled (default).

    max
     - the  maximum  size  of  the  receive  buffer  used  by  each  TCP  socket. This  value  does  not  override  the 
       global  net.core.rmem_max. This is not used to limit the size of the receive buffer declared using SO_RCVBUF on
       a socket. The default  value of 87380*2 bytes is lowered to 87380 in low-memory systems.
==================================
Questions

1. When default value is 87380 for receive buffer , why they advertise window size of much lower values ?
2. The default max for receive ( man pages) says 87,380 * 2 but the sysctl says 4194304 bytes ? ( as per RFC,the max receive buffer determines the scale factor.)
3. Is there a way to determine socket buffer size of a TCP connection dynamically ? I looked at /proc/net/tcp and /proc/net/sockstat
which gives socket location, references and overall socket size, number of TCP connections respectively.

Thanks
Padmanabhan





-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Iscsitarget-devel mailing list
Iscsitarget-devel@...
https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel

[Padmanabhan]

Any suggestions or pointers to resources would be great.

thanks
Padmanabhan

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Iscsitarget-devel mailing list
Iscsitarget-devel@...
https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel

[Jeff Waller]

Hi Padmanabhan,

Here's a reference for you on TCP in general, but obviously it's not  
written
with ISCSI specifically in mind:

http://acs.lbl.gov/TCP-tuning/linux.html

After having read that, I modified my systems properties in the  
following way:

# increase TCP max buffer size setable using setsockopt()
net.core.rmem_default = 1048576
net.core.wmem_default = 1048576
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
# set max to at least 4MB, or higher if you use very high BDP paths
net.ipv4.tcp_mem = 1048576 1048576 16777216
net.ipv4.tcp_rmem = 1048576 1048576 16777216
net.ipv4.tcp_wmem = 1048576 1048576 16777216
# don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
# recommended to increase this for 1000 BT or higher
net.core.netdev_max_backlog = 2500
# Setting the Congestion Avoidance Algorithm
net.ipv4.tcp_congestion_control=htcp


Here's the relevant portion of my sysctl.conf.  Did it help?  Well,  
like you,
I'm just beginning this.  I can report some numbers and maybe some one
can say if it's good or not?  I made the adjustments with the thought  
that
bigger buffers are better --- aren't they?

First off  I'll describe my setup:

The target is a linux host using software Raid-5 across 5 750 GB SATA
disks 2.6.25 kernel with (as can be see above) with all the the new
congestion avoidance algorithms compiled in as modules (see the
above reference).  It exports it's bandwidth in 2 ways:

1) Using dual gigabit ethernet controllers in an LACP bundle connected
to a Cisco 2970 switch.  Note that using LACP will only help when  
multiple
initiators are communicating as the 802.3ad spec requires the same 2
endpoints to use the same communication path.

2) Using a single 10/100 controller connected to the same 2970 switch,
but using a different VLAN.

The gigabit network has jumbo packets (MTU 9000) enabled.

Initiators connect to the appropriate controller based on their network
interface manually (using a different IP address).

Here's the network setup (note syntax is gentoo linux) /etc/conf.d/net:

config_bond0="192.168.0.2 netmask 255.255.255.0"
slaves_bond0="eth0 eth1"
mtu_bond0="9000"
txqueuelen_bond0="1000"
config_eth2="192.168.1.31 netmask 255.255.255.0"
routes_eth2="default gw 192.168.1.1"

-Jeff



On Jun 25, 2008, at 7:23 PM, Padmanabhan wrote:

> Any suggestions or pointers to resources would be great.
>
> thanks
> Padmanabhan
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php_______________________________________________
> Iscsitarget-devel mailing list
> Iscsitarget-devel@...
> https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Iscsitarget-devel mailing list
Iscsitarget-devel@...
https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel