From dmvasil at ornl.gov Fri Nov 17 08:26:31 2006 From: dmvasil at ornl.gov (David Vasil) Date: Fri, 17 Nov 2006 08:26:31 -0500 Subject: [osiris-devel] fixing the filters implementation Message-ID: <455DB887.8060501@ornl.gov> I've been looking into why the filters always get rearranged when you do an edit-filters and it turns out this is the way it should work in the current implementation. Since Osiris stores the filters in a BDB, the BDB choses what order to store the items in based on its own sorting algorithm. Looking at how the osirismd uses the filters during scans, I see no benefit in storing the filters in a database. In md_scan.c the OsirisMD does a sequential lookup on every filter in the BDB to see if it matches the current input. To me this seems like it would create more overhead than just a flat file. Changing this to be a flat text file would fix the problem with the filters being rearranged between edits and it may improve performance a small amount on the MD. What are other's thoughts on this? -- -dave From fischerdk at fidoki.com Sat Nov 18 18:00:52 2006 From: fischerdk at fidoki.com (Douglas K. Fischer) Date: Sat, 18 Nov 2006 18:00:52 -0500 Subject: [osiris-devel] Re: fixing the filters implementation In-Reply-To: <455DB887.8060501@ornl.gov> References: <455DB887.8060501@ornl.gov> Message-ID: <455F90A4.7020509@fidoki.com> David Vasil wrote: > I've been looking into why the filters always get rearranged when you do > an edit-filters and it turns out this is the way it should work in the > current implementation. Since Osiris stores the filters in a BDB, the > BDB choses what order to store the items in based on its own sorting > algorithm. > > Looking at how the osirismd uses the filters during scans, I see no > benefit in storing the filters in a database. In md_scan.c the OsirisMD > does a sequential lookup on every filter in the BDB to see if it matches > the current input. To me this seems like it would create more overhead > than just a flat file. > > Changing this to be a flat text file would fix the problem with the > filters being rearranged between edits and it may improve performance a > small amount on the MD. What are other's thoughts on this? > I'm all for this. The whole rearranging business of BDB is the reason I created the --import-filters function. That way I could maintain a text file for the filters for ease of management and then just import it into the DB and let it do whatever it wanted with them. Eliminating the filter DB would simplify things greatly. FWIW, Doug From dmvasil at ornl.gov Mon Nov 20 08:39:12 2006 From: dmvasil at ornl.gov (David Vasil) Date: Mon, 20 Nov 2006 08:39:12 -0500 Subject: [osiris-devel] Re: fixing the filters implementation In-Reply-To: <455F90A4.7020509@fidoki.com> References: <455DB887.8060501@ornl.gov> <455F90A4.7020509@fidoki.com> Message-ID: <4561B000.1050004@ornl.gov> Douglas K. Fischer wrote: > David Vasil wrote: >> I've been looking into why the filters always get rearranged when you do >> an edit-filters and it turns out this is the way it should work in the >> current implementation. Since Osiris stores the filters in a BDB, the >> BDB choses what order to store the items in based on its own sorting >> algorithm. >> >> Looking at how the osirismd uses the filters during scans, I see no >> benefit in storing the filters in a database. In md_scan.c the OsirisMD >> does a sequential lookup on every filter in the BDB to see if it matches >> the current input. To me this seems like it would create more overhead >> than just a flat file. >> >> Changing this to be a flat text file would fix the problem with the >> filters being rearranged between edits and it may improve performance a >> small amount on the MD. What are other's thoughts on this? >> > I'm all for this. The whole rearranging business of BDB is the reason I > created the --import-filters function. That way I could maintain a text > file for the filters for ease of management and then just import it into > the DB and let it do whatever it wanted with them. Eliminating the > filter DB would simplify things greatly. I talked with a few programmer friends, and they seem to think that it would be a better idea to leave the filters in a BDB and store a master file on the MD (much like how mail aliases are generated for MTA's). I almost have this implemented, we will see how it works out. -- -dave From dmvasil at ornl.gov Tue Nov 21 16:40:25 2006 From: dmvasil at ornl.gov (David Vasil) Date: Tue, 21 Nov 2006 16:40:25 -0500 Subject: [osiris-devel] Re: fixing the filters implementation In-Reply-To: <4561B000.1050004@ornl.gov> References: <455DB887.8060501@ornl.gov> <455F90A4.7020509@fidoki.com> <4561B000.1050004@ornl.gov> Message-ID: <45637249.1070505@ornl.gov> David Vasil wrote: > Douglas K. Fischer wrote: >> David Vasil wrote: >>> I've been looking into why the filters always get rearranged when you do >>> an edit-filters and it turns out this is the way it should work in the >>> current implementation. Since Osiris stores the filters in a BDB, the >>> BDB choses what order to store the items in based on its own sorting >>> algorithm. >>> >>> Looking at how the osirismd uses the filters during scans, I see no >>> benefit in storing the filters in a database. In md_scan.c the OsirisMD >>> does a sequential lookup on every filter in the BDB to see if it matches >>> the current input. To me this seems like it would create more overhead >>> than just a flat file. >>> >>> Changing this to be a flat text file would fix the problem with the >>> filters being rearranged between edits and it may improve performance a >>> small amount on the MD. What are other's thoughts on this? >>> >> I'm all for this. The whole rearranging business of BDB is the reason I >> created the --import-filters function. That way I could maintain a text >> file for the filters for ease of management and then just import it into >> the DB and let it do whatever it wanted with them. Eliminating the >> filter DB would simplify things greatly. > > I talked with a few programmer friends, and they seem to think that it > would be a better idea to leave the filters in a BDB and store a master > file on the MD (much like how mail aliases are generated for MTA's). I > almost have this implemented, we will see how it works out. I have this written against the current 4.2.2 release (the md_compare.c patch is also added into this patch). I have not applied this to the current svn trunk yet as I would like to have it tested a little before committing it. I tested this on Linux (Ubuntu, RHEL, Fedora), OpenBSD (4.0), and Windows (the code works as expected, but filters in general dont appear to work under Windows [have they ever? 4.2.2 without the patches behave the same way]). I added in a function to md_filter.c to create the filter file if it does not exist already by dumping the current filter database to the file. This will prevent people's filters from being lost after upgrading to this code. I'll be running this code on my system for a while to see how it works out. If others can run the code or look over it for obvious mistakes, please let me know. -- -Dave -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: osiris-4.2.2-filter_file-md_compare.patch Url: http://lists.shmoo.com/pipermail/osiris-devel/attachments/20061121/2078c9e6/attachment.txt From dmvasil at ornl.gov Wed Nov 29 14:57:54 2006 From: dmvasil at ornl.gov (David Vasil) Date: Wed, 29 Nov 2006 14:57:54 -0500 Subject: [osiris-devel] Re: fixing the filters implementation In-Reply-To: <45637249.1070505@ornl.gov> References: <455DB887.8060501@ornl.gov> <455F90A4.7020509@fidoki.com> <4561B000.1050004@ornl.gov> <45637249.1070505@ornl.gov> Message-ID: <456DE642.6040104@ornl.gov> David Vasil wrote: > I have this written against the current 4.2.2 release (the md_compare.c > patch is also added into this patch). I have not applied this to the > current svn trunk yet as I would like to have it tested a little before > committing it. I tested this on Linux (Ubuntu, RHEL, Fedora), OpenBSD > (4.0), and Windows (the code works as expected, but filters in general > dont appear to work under Windows [have they ever? 4.2.2 without the > patches behave the same way]). > > I added in a function to md_filter.c to create the filter file if it > does not exist already by dumping the current filter database to the > file. This will prevent people's filters from being lost after > upgrading to this code. > > I'll be running this code on my system for a while to see how it works > out. If others can run the code or look over it for obvious mistakes, > please let me know. I've been testing this on my system without a problem for about a week now. If noone has any objections, I'll commit this to the subversion trunk and it will be part of the next release. -- -dave From dmvasil at ornl.gov Wed Nov 29 17:04:03 2006 From: dmvasil at ornl.gov (David Vasil) Date: Wed, 29 Nov 2006 17:04:03 -0500 Subject: [osiris-devel] strange race condition in osirisctl.c Message-ID: <456E03D3.9070703@ornl.gov> In src/libosirisctl/osirisctl.c:ctl_get_database() at the break; on line 3808, the status should be set to TRUE when the function returns, however on some systems a FALSE is returned to the calling function src/cli/osiris.c:process_print_db() at the if statement on line 2867. This causes the CLI print-db to fail even though the temporary database was correctly written to the CLI host as shown by trying to diff the db.temp and the original DB on the management host. When debugging this issue, I found that putting a sleep(1); right before the offending break in osirisctl.c on line 3808 fixes the problem. Can anyone provide insight as to why this workaround works, and maybe provide a better fix? This condition exists on some Debian systems as reported by Ruben Puettmann and I was able to recreate the error on a Fedora Core 5 i386 virtual machine. Thank you! -- -dave