[Openvas-devel] New NVT server side cache to overcome NVT size limits

[Jan-Oliver Wagner]

Hello,

I was unhappy with the current cache for the major reason that
the cache as designed by the Nessus people limits NVTs in various ways.
E.g. length of description, number of dependencies, number of CVEs etc.
are all limited.

The current cache works with fixed sizes, and just dumps binary
memory blocks to/from filesystem.

I've designed a keyfile approach where each cache file is a keyfile
corresponding to a NVT.
The limits of the keyfiles are far beyond what we have currently.

I've used the nvti.c module which you find in current openvas-libraries
of trunk and did some measurements:

Applied for: openvas-plugins rev 3609
 + a couple of personal scripts
 = 12028 scripts

Patch for measurement with time command:

Index: openvasd/pluginload.c
===================================================================
--- openvasd/pluginload.c       (Revision 3609)
+++ openvasd/pluginload.c       (Arbeitskopie)
@@ -219,6 +219,7 @@
   if ( be_quiet == 0 )
          printf("\rAll plugins loaded                                   \n");

+exit(0);
   return plugins;
 }

Results:

Keyfile solution: Initial start with empty cache:
time openvasd
real    4m4.297s
user    1m15.853s
sys     1m16.549s

Keyfile solution: Start with cache:
time openvasd
real    0m6.960s
user    0m5.892s
sys     0m0.700s

Keyfile solution: Size of Cache:
du -sh .
48M     .

Old cache version with same NVTs:

Current binary cache: Start without cache:
time openvasd
real    5m42.230s
user    1m15.465s
sys     1m7.948s

Current binary cache: Start with cache:
real    0m3.453s
user    0m2.808s
sys     0m0.496s

Current binary cache: Cache size:
du -sh .
142M    .

Summary:
* Building the cache anew duration is almost the same
* Loading existing cache duration doubles with keyfiles, but still at ~2000 
NVTs per second
* Harddisk consumption drops to ~1/3 of the current solution
* There is still more opportunities to increase speed, as I did not optimize
   for speed yet.
* No limits for description, CVEs, dependencies, ... 

This patch could be integrated after we branch openvas-libraries for 2.1-
series.

Opinions?
Worth a Change Request?

Best

	Jan

-- 
Dr. Jan-Oliver Wagner |  ++49-541-335084-0  |  http://www.greenbone.net/
Greenbone Networks GmbH, Neuer Graben 17, 49074 Osnabrück
AG Osnabrück, HR B 202460 | Geschäftsführer: Lukas Grunwald, Dr. Jan-Oliver 
Wagner