Re: [rrd-users] building a huge rrd / any limitations?

[Sebastian Harl]

Hi Karl,

(Not going to comment on the usefulness of having such large RRD's. ;-))

On Wed, Jun 11, 2008 at 03:14:18AM +0200, Karl Fischer wrote:
> I'm planning to create a really huge rrd with 5-10 million rows
> holding about 100-200 entries each, so the complete rrd will reach
> a size of 5 .. 20 GB ...
> Are there any problems to expect when creating and using an rrd
> that size? Any limits to hit (like 32bit file pointers or so)?

Yes, currently RRDtool is basically unaware of large files. Some 64 bits
architectures might happen to support that anyway but I don't think
that's guaranteed.

Attached to this E-mail you can find a patch that adds _very_ _basic_
large file support. Please note though, that this patch won't help you
much in your current situation as other parts of RRDtool still cannot
handle large files (see e.g. one of my replies to Debian bug #451852
[1]). However, that patch is the base which is necessary to further go
into that direction. See the patch description for further details.

For that patch to be effective, it is necessary, that all source files
that work with files directly include rrd_config.h (either directly or
by including rrd_tool.h) (which is currently the case). Also, fseek()
and ftell() should not be used but replaced with the newly added macros
FSEEK and FTELL.

Cheers,
Sebastian

[1] http://bugs.debian.org/451852#49

--
Sebastian "tokkee" Harl +++ GnuPG-ID: 0x8501C7FC +++ http://tokkee.org/

Those who would give up Essential Liberty to purchase a little Temporary
Safety, deserve neither Liberty nor Safety.         -- Benjamin Franklin



_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

[Sebastian Harl]

Hi,

On Wed, Jun 11, 2008 at 11:34:28PM +0200, Sebastian Harl wrote:
> Attached to this E-mail you can find a patch that adds _very_ _basic_
> large file support.

D'oh! Now, it really is attached to this E-mail ;-)

Cheers,
Sebastian

--
Sebastian "tokkee" Harl +++ GnuPG-ID: 0x8501C7FC +++ http://tokkee.org/

Those who would give up Essential Liberty to purchase a little Temporary
Safety, deserve neither Liberty nor Safety.         -- Benjamin Franklin


From 4713f65cf958a7abc5a0fb35aa20d35699a45922 Mon Sep 17 00:00:00 2001
From: Sebastian Harl <sh@...>
Date: Wed, 11 Jun 2008 22:58:39 +0200
Subject: [PATCH] Added basic large file support.

This patch makes RRDtool aware of large files by adding support to access them
thru the libc interface. This is done by adding the AC_SYS_LARGEFILE macro to
configure.ac which takes care of all the required magic to achieve that. This
macro basically ensures that the off_t type is at least 64 bits wide and takes
measures to transparently choose libc functions that are aware of 64 bits file
sizes.

Two functions, fseek() and ftell(), require special care to support large
files - fseeko() and ftello() have been introduced for that purpose. Those
latter functions are now used, if available.
---
 program/configure.ac   |    5 +++++
 program/src/rrd_cgi.c  |    7 ++++---
 program/src/rrd_tool.h |   12 ++++++++++++
 3 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/program/configure.ac b/program/configure.ac
index 7a15b5c..6c86841 100644
--- a/program/configure.ac
+++ b/program/configure.ac
@@ -11,6 +11,11 @@ dnl the official version number is
 dnl a.b.c
 AC_INIT([rrdtool],[1.3.0])
 
+dnl check for large file support
+AC_SYS_LARGEFILE
+AC_FUNC_FSEEKO
+AC_CHECK_FUNC(ftello)
+
 dnl for testing a numberical version number comes handy
 dnl the released version are
 dnl a.bccc
diff --git a/program/src/rrd_cgi.c b/program/src/rrd_cgi.c
index d4e5723..2066d8c 100644
--- a/program/src/rrd_cgi.c
+++ b/program/src/rrd_cgi.c
@@ -416,12 +416,13 @@ static int readfile(
         } while (c != '\n' && !feof(input));
     }
     if (strcmp("-", file_name)) {
-        fseek(input, 0, SEEK_END);
+        FSEEK(input, 0, SEEK_END);
+
         /* have extra space for detecting EOF without realloc */
-        totalcnt = (ftell(input) + 1) / sizeof(char) - offset;
+        totalcnt = (FTELL(input) + 1) / sizeof(char) - offset;
         if (totalcnt < MEMBLK)
             totalcnt = MEMBLK;  /* sanitize */
-        fseek(input, offset * sizeof(char), SEEK_SET);
+        FSEEK(input, offset * sizeof(char), SEEK_SET);
     }
     if (((*buffer) = (char *) malloc((totalcnt + 4) * sizeof(char))) == NULL) {
         perror("Allocate Buffer:");
diff --git a/program/src/rrd_tool.h b/program/src/rrd_tool.h
index 0be66e4..fa1b605 100644
--- a/program/src/rrd_tool.h
+++ b/program/src/rrd_tool.h
@@ -18,6 +18,18 @@ extern    "C" {
 
 #include "rrd.h"
 
+#if HAVE_FSEEKO
+#define FSEEK fseeko
+#else
+#define FSEEK fseek
+#endif
+
+#if HAVE_FTELLO
+#define FTELL ftello
+#else
+#define FTELL ftell
+#endif
+
 #if defined(_WIN32) && !defined(__CYGWIN__) && !defined(__CYGWIN32__)
 
 /* Win32 only includes */
--
1.5.6.rc2



_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

[Karl Fischer-3]

> [1] http://bugs.debian.org/451852#49

Sebastian,

just studied the thread ...
Well, creating a database that spans ~1150 years not only breaks LONG,
it also breaks the current UNIX time function, that allows for a range
of ~68 years (01.01.1970 - 18.01.2038) ...

;-)

- Karl




_______________________________________________
rrd-developers mailing list
rrd-developers@...
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers