Numerous "CIFS VFS: Send error" messages in 2.6.25.3

[Gautam Iyer-4]

Hi All,

I was using "mount -t smbfs" on Linux-2.6.23 for a long time with no
problems. When I recently upgraded to CIFS + Linux-2.6.25, I ran into a
few issues.

My Samba server is a Vantec networked hard disk. I mount it via

    mount -t cifs //andurin/PUBLIC /tmp/andurin -o nounix,guest

Now I can copy / move files just fine. But when I try and do something
fancy, I get numerous errors. Usually at this point my log file is
clutter with numerous CIFS VFS error messages. A few of the log lines
are:

    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
    May 12 18:56:04 mordor CIFS VFS: No response to cmd 47 mid 67
    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote 0
    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote -511544036
    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote -511544036
    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
    May 12 18:56:04 mordor CIFS VFS: No response to cmd 47 mid 78

Again, everything (seems) to work fine for simple file operations (cp
-R, mv, rm, head, tail). The operations that mess up are things like
mke2fs, zpool create, rsync on really large and similar directory trees.

My NAS server is working fine since I can access it file from my wifes
MAC. But again, the Mac only does simple file operations (cp, rm,
generating thumbnails).

Would you know what's going on? Or what I can do to work around this
problem?

Thanks,

GI

--
Microsoft broke Volkswagen's world record: Volkswagen only made 22
million bugs!


_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Shirish Pargaonkar]

On 5/12/08, Gautam Iyer <gi1242+samba@...> wrote:

> Hi All,
>
> I was using "mount -t smbfs" on Linux-2.6.23 for a long time with no
> problems. When I recently upgraded to CIFS + Linux-2.6.25, I ran into a
> few issues.
>
> My Samba server is a Vantec networked hard disk. I mount it via
>
>    mount -t cifs //andurin/PUBLIC /tmp/andurin -o nounix,guest
>
> Now I can copy / move files just fine. But when I try and do something
> fancy, I get numerous errors. Usually at this point my log file is
> clutter with numerous CIFS VFS error messages. A few of the log lines
> are:
>
>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
>    May 12 18:56:04 mordor CIFS VFS: No response to cmd 47 mid 67
>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote 0
>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote -511544036
>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote -511544036
>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
>    May 12 18:56:04 mordor CIFS VFS: No response to cmd 47 mid 78
>
> Again, everything (seems) to work fine for simple file operations (cp
> -R, mv, rm, head, tail). The operations that mess up are things like
> mke2fs, zpool create, rsync on really large and similar directory trees.
>
> My NAS server is working fine since I can access it file from my wifes
> MAC. But again, the Mac only does simple file operations (cp, rm,
> generating thumbnails).
>
> Would you know what's going on? Or what I can do to work around this
> problem?
>
> Thanks,
>
> GI
>
> --
> Microsoft broke Volkswagen's world record: Volkswagen only made 22
> million bugs!
>
> _______________________________________________
> linux-cifs-client mailing list
> linux-cifs-client@...
> https://lists.samba.org/mailman/listinfo/linux-cifs-client
>
>
>

I am trying to work on this problem.  Try increasing the send
(sk_sndbuf) and receive
(sk_rcvbuf) buffer sizes of socket  from 200K and 140K respectively to
512K or 1M
in fs/cifs/connect.c in function ipv4_connect.

What is the output of command
sysctl -a | grep tcp | grep mem

I am not sure whether cifs should be setting these values of sndbuf and
rcvbuf of a socket and instead let tcp auto-tuning take care of the sizes but
I think auto-tuning on send buffer side is not working, recieve buffer
side it does.


Regards,

Shirish
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Gautam Iyer-4]

On Mon, May 12, 2008 at 10:29:12PM -0500, Shirish Pargaonkar wrote:

>> I was using "mount -t smbfs" on Linux-2.6.23 for a long time with no
>> problems. When I recently upgraded to CIFS + Linux-2.6.25, I ran into a
>> few issues.
>>
>> My Samba server is a Vantec networked hard disk. I mount it via
>>
>>    mount -t cifs //andurin/PUBLIC /tmp/andurin -o nounix,guest
>>
>> Now I can copy / move files just fine. But when I try and do something
>> fancy, I get numerous errors. Usually at this point my log file is
>> clutter with numerous CIFS VFS error messages. A few of the log lines
>> are:
>>
>>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
>>    May 12 18:56:04 mordor CIFS VFS: No response to cmd 47 mid 67
>>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote 0
>>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote -511544036
>>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
>>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote -511544036
>>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
>>    May 12 18:56:04 mordor CIFS VFS: No response to cmd 47 mid 78
>>
> I am trying to work on this problem.  Try increasing the send
> (sk_sndbuf) and receive (sk_rcvbuf) buffer sizes of socket  from 200K
> and 140K respectively to 512K or 1M in fs/cifs/connect.c in function
> ipv4_connect.

Thanks for the quick response. I did this, and seem to get the same
messages in my syslog, and the same error message from fancy operations
(e.g. mke2fs). (I'm also assuming that modprobe -rv cifs, and then
modprobe -iv cifs was enough for this change to take effect).

> What is the output of command
> sysctl -a | grep tcp | grep mem

It was

    net.ipv4.tcp_mem = 96672        128896  193344
    net.ipv4.tcp_wmem = 4096        16384   4124672
    net.ipv4.tcp_rmem = 4096        87380   4124672

Plus a few error messages about ipv6.

Let me know if there's anything else you want me to test, or if there is
anything else I can do to help fix this,

Thanks

GI
 
--
'Common' Proof Techniques:
5. Proof by Omission -- "The reader may easily supply the details"


_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Shirish Pargaonkar]

On 5/13/08, Gautam Iyer <gi1242+samba@...> wrote:

> On Mon, May 12, 2008 at 10:29:12PM -0500, Shirish Pargaonkar wrote:
>
> >> I was using "mount -t smbfs" on Linux-2.6.23 for a long time with no
> >> problems. When I recently upgraded to CIFS + Linux-2.6.25, I ran into a
> >> few issues.
> >>
> >> My Samba server is a Vantec networked hard disk. I mount it via
> >>
> >>    mount -t cifs //andurin/PUBLIC /tmp/andurin -o nounix,guest
> >>
> >> Now I can copy / move files just fine. But when I try and do something
> >> fancy, I get numerous errors. Usually at this point my log file is
> >> clutter with numerous CIFS VFS error messages. A few of the log lines
> >> are:
> >>
> >>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
> >>    May 12 18:56:04 mordor CIFS VFS: No response to cmd 47 mid 67
> >>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote 0
> >>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote -511544036
> >>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
> >>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote -511544036
> >>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
> >>    May 12 18:56:04 mordor CIFS VFS: No response to cmd 47 mid 78
> >>
> > I am trying to work on this problem.  Try increasing the send
> > (sk_sndbuf) and receive (sk_rcvbuf) buffer sizes of socket  from 200K
> > and 140K respectively to 512K or 1M in fs/cifs/connect.c in function
> > ipv4_connect.
>
> Thanks for the quick response. I did this, and seem to get the same
> messages in my syslog, and the same error message from fancy operations
> (e.g. mke2fs). (I'm also assuming that modprobe -rv cifs, and then
> modprobe -iv cifs was enough for this change to take effect).
>
> > What is the output of command
> > sysctl -a | grep tcp | grep mem
>
> It was
>
>    net.ipv4.tcp_mem = 96672        128896  193344
>    net.ipv4.tcp_wmem = 4096        16384   4124672
>    net.ipv4.tcp_rmem = 4096        87380   4124672
>
> Plus a few error messages about ipv6.
>
> Let me know if there's anything else you want me to test, or if there is
> anything else I can do to help fix this,
>
> Thanks
>
> GI
>
> --
> 'Common' Proof Techniques:
> 5. Proof by Omission -- "The reader may easily supply the details"
>
> _______________________________________________
> linux-cifs-client mailing list
> linux-cifs-client@...
> https://lists.samba.org/mailman/listinfo/linux-cifs-client
>
>
>

Gautam,

You can try taking out the setting the send and receive socket buffer
sizes altogether.
Let tcp auto-tuning take care of the socket buffer sizes.
What I found out is with a stress testssuites like fsstress, send
buffer does not
tune (increase in size) but with your test load, may be it will work.
Error -11 is EAGAIN and -28 is ENOSPC, so I am suspecting it has to do with
socket buffers.
So a patch like this might work.

Regards,

Shirish


diff --git a/fs/cifs/connect.c b/fs/cifs/connect.c
index 791ca5c..c9a2090 100644
--- a/fs/cifs/connect.c
+++ b/fs/cifs/connect.c
@@ -1550,11 +1550,6 @@ ipv4_connect(struct sockaddr_in *psin_se
                 (*csocket)->sk->sk_sndbuf,
                 (*csocket)->sk->sk_rcvbuf, (*csocket)->sk->sk_rcvtimeo));
        (*csocket)->sk->sk_rcvtimeo = 7 * HZ;
-       /* make the bufsizes depend on wsize/rsize and max requests */
-       if ((*csocket)->sk->sk_sndbuf < (200 * 1024))
-               (*csocket)->sk->sk_sndbuf = 200 * 1024;
-       if ((*csocket)->sk->sk_rcvbuf < (140 * 1024))
-               (*csocket)->sk->sk_rcvbuf = 140 * 1024;

        /* send RFC1001 sessinit */
        if (psin_server->sin_port == htons(RFC1001_PORT)) {
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Gautam Iyer-4]

On Tue, May 13, 2008 at 09:28:09AM -0500, Shirish Pargaonkar wrote:

>>>> I was using "mount -t smbfs" on Linux-2.6.23 for a long time with no
>>>> problems. When I recently upgraded to CIFS + Linux-2.6.25, I ran into a
>>>> few issues.
>>>>
>>>> My Samba server is a Vantec networked hard disk. I mount it via
>>>>
>>>>    mount -t cifs //andurin/PUBLIC /tmp/andurin -o nounix,guest
>>>>
>>>> Now I can copy / move files just fine. But when I try and do something
>>>> fancy, I get numerous errors. Usually at this point my log file is
>>>> clutter with numerous CIFS VFS error messages. A few of the log lines
>>>> are:
>>>>
>>>>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
>>>>    May 12 18:56:04 mordor CIFS VFS: No response to cmd 47 mid 67
>>>>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote 0
>>>>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote -511544036
>>>>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
>>>>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -11, wrote -511544036
>>>>    May 12 18:56:04 mordor CIFS VFS: Write2 ret -28, wrote 0
>>>>    May 12 18:56:04 mordor CIFS VFS: No response to cmd 47 mid 78
>>>>
>>> I am trying to work on this problem.  Try increasing the send
>>> (sk_sndbuf) and receive (sk_rcvbuf) buffer sizes of socket  from 200K
>>> and 140K respectively to 512K or 1M in fs/cifs/connect.c in function
>>> ipv4_connect.
>>
>> Thanks for the quick response. I did this, and seem to get the same
>> messages in my syslog, and the same error message from fancy operations
>> (e.g. mke2fs). (I'm also assuming that modprobe -rv cifs, and then
>> modprobe -iv cifs was enough for this change to take effect).
>>
>>> What is the output of command
>>> sysctl -a | grep tcp | grep mem
>>
>> It was
>>
>>    net.ipv4.tcp_mem = 96672        128896  193344
>>    net.ipv4.tcp_wmem = 4096        16384   4124672
>>    net.ipv4.tcp_rmem = 4096        87380   4124672
>>
>> Plus a few error messages about ipv6.
>
> You can try taking out the setting the send and receive socket buffer
> sizes altogether. Let tcp auto-tuning take care of the socket buffer
> sizes. What I found out is with a stress testssuites like fsstress,
> send buffer does not tune (increase in size) but with your test load,
> may be it will work. Error -11 is EAGAIN and -28 is ENOSPC, so I am
> suspecting it has to do with socket buffers. So a patch like this
> might work.

Hi Shirish,

I tried this with no success! I got a message

    CIFS VFS: Write2 ret -28, wrote 0
    CIFS VFS: Write2 ret -11, wrote -893664708
    CIFS VFS: Write2 ret -28, wrote 0
    CIFS VFS: No response to cmd 47 mid 23

Then got numerous repetitions of the above block with different numbers
after "mid" in the last line. Towards the end I get the following:

    CIFS VFS: No response to cmd 46 mid 170
    CIFS VFS: Send error in read = -11
    CIFS VFS: Write2 ret -28, wrote 0
    CIFS VFS: No response for cmd 50 mid 176
    CIFS VFS: Send error in read = -5
    CIFS VFS: No response to cmd 4 mid 184
    CIFS VFS: Send error in Close = -11

Thanks again for both your quick responses and suggestions,

GI

PS: Perhaps I should also mention -- on 2.6.23 with smbfs I would get
    really low read speeds (300kb/s). With cifs on 2.6.25 I can copy
    large files from my Samba server (NAS) to my laptop with much higher
    speeds (generally 1.2mb/s, but sometimes up to 2 mb/s). These speeds
    are comparable to the speeds I get when getting the files via ftp,
    and my wife's mac seems to perform file operations on the Samba
    server at similar speeds with no problems. I don't mind the lower
    speeds, so long as it's reliable :).

--
Alternative definitions of terms from Math Lectures:
CLEARLY: I don't want to write down all the "in-between" steps.


_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client