DO NOT REPLY [Bug 3491] New: throttle disk IO during filelist/directory parsing

[bugzilla-daemon-7]

https://bugzilla.samba.org/show_bug.cgi?id=3491

           Summary: throttle disk IO during filelist/directory parsing
           Product: rsync
           Version: 2.6.4
          Platform: All
               URL: http://vilius.multiply.com/video/item/10
        OS/Version: Linux
            Status: NEW
          Severity: enhancement
          Priority: P3
         Component: core
        AssignedTo: wayned@...
        ReportedBy: vilius@...
         QAContact: rsync-qa@...


rsync was bringing our webserver to crawl while during file list generation.
since the job wasn't time critical, i made it be less aggressive during this
step - a rather trivial change to microsleep between each readdir().

--slow-down=100 will usleep() for 1000usec (microseconds) before each readdir.
if i'm not mistaken with 10k directories that'd be ~10second of sleep.


I've seen people try to do this using --bwlimit and/or loop checking loadavg
and sending sigstop/sigcont.
re: http://lists.samba.org/archive/rsync/2004-February/008651.html

not the best fix for disk cache poisoning but still might give a some time to
breathe.

sorry, patches are not for the latest ver..


--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[bugzilla-daemon-7]

https://bugzilla.samba.org/show_bug.cgi?id=3491





------- Comment #1 from vanes002@...  2006-02-08 02:22 MST -------
I like this idea.  I, too, have seen degradation in response times when rsync
starts on a large hierarchy.  Using nice didn't help (much).

Wouldn't it be better to count up readdir() calls and then sleep for a longer
time when a threshold is reached?  Fewer system calls that way.  For example,
add the slow-down value to a counter and when the counter passes 100,000 then
sleep for 100,000 microseconds (0.1 second) and subtract 100,000 from the
counter.

BTW, in your example, did you mean --slow-down=1000 (not 100) for 1000
microsecs?


--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html