[flow-tools] Potential speedup of sequential reading of large files

[Jos Backus]

Fyi:

On Solaris 2.7 and up one can turn on directio on a file descriptor using

    directio(0, DIRECTIO_ON)

See directio(3C) for details.

On FreeBSD 4.3 (I think) and up one can instead use

    open(..., O_DIRECT)

See open(2) for details.

The basic idea is to tell the OS not to keep file pages in the cache that are
not going to be needed shortly afterwards, as is the case when processing a
file sequentially. The idea is sort of similar in spirit to what madvise's
MADV_SEQUENTIAL flag does for accessing mmap'ed memory.

On Solaris this gives us roughly a 10%-30% speed increase in an application
that sequentially reads a large flow file (500MB average); I have not tested
FreeBSD but I would expect to see a similar improvement.

-- 
Jos Backus                 _/  _/_/_/        Santa Clara, CA
                          _/  _/   _/
                         _/  _/_/_/             
                    _/  _/  _/    _/
josb@cncdsl.com     _/_/   _/_/_/            use Std::Disclaimer;