[osiris-devel] Re: fixing the filters implementation

[David Vasil]

David Vasil wrote:
> Douglas K. Fischer wrote:
>> David Vasil wrote:
>>> I've been looking into why the filters always get rearranged when you do
>>> an edit-filters and it turns out this is the way it should work in the
>>> current implementation.  Since Osiris stores the filters in a BDB, the
>>> BDB choses what order to store the items in based on its own sorting
>>> algorithm.
>>>
>>> Looking at how the osirismd uses the filters during scans, I see no
>>> benefit in storing the filters in a database.  In md_scan.c the OsirisMD
>>> does a sequential lookup on every filter in the BDB to see if it matches
>>> the current input.  To me this seems like it would create more overhead
>>> than just a flat file.
>>>
>>> Changing this to be a flat text file would fix the problem with the
>>> filters being rearranged between edits and it may improve performance a
>>> small amount on the MD.  What are other's thoughts on this?
>>>   
>> I'm all for this. The whole rearranging business of BDB is the reason I
>> created the --import-filters function. That way I could maintain a text
>> file for the filters for ease of management and then just import it into
>> the DB and let it do whatever it wanted with them. Eliminating the
>> filter DB would simplify things greatly.
> 
> I talked with a few programmer friends, and they seem to think that it
> would be a better idea to leave the filters in a BDB and store a master
> file on the MD (much like how mail aliases are generated for MTA's).  I
> almost have this implemented, we will see how it works out.

I have this written against the current 4.2.2 release (the md_compare.c
patch is also added into this patch).  I have not applied this to the
current svn trunk yet as I would like to have it tested a little before
committing it.  I tested this on Linux (Ubuntu, RHEL, Fedora), OpenBSD
(4.0), and Windows (the code works as expected, but filters in general
dont appear to work under Windows [have they ever? 4.2.2 without the
patches behave the same way]).

I added in a function to md_filter.c to create the filter file if it
does not exist already by dumping the current filter database to the
file.  This will prevent people's filters from being lost after
upgrading to this code.

I'll be running this code on my system for a while to see how it works
out.  If others can run the code or look over it for obvious mistakes,
please let me know.

-- 
-Dave

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: osiris-4.2.2-filter_file-md_compare.patch
Url: http://lists.shmoo.com/pipermail/osiris-devel/attachments/20061121/2078c9e6/attachment.txt