A possible solution to the billions-of-facts-in-memory is to support a sort of fact paging system. The idea is that the rule matching algorithm can optimize for space as well as, or instead of, speed, by matching rules relevant to a given set of fact types together, and paging out the fact types it doesn't expect to have to worry about for a while.
if we had parameters matching the types:
- org.example.ZipCode
- org.example.Animal
- org.example.Vegetable
The WM could safely offload to some persistent storage all of the other types of facts until such a time as a rule was created that was relevant to those facts.
Even better, it could build the matching network such that it could effectively infer when a class of facts would be logically unreachable, take the following example:
- parameters: ZipCode, Animal
- parameters: ZipCode, Vegetable
- parameters: Animal, Vegetable
- parameters: Animal, Animal
In this case if there are no vegetables asserted into the WM there can be no rule matching ZipCode matched, period, so all zipcodes can be safely offloaded.
If a vegetable is asserted into the working memory, the zipcodes will need to be retrieved in order for the [ZipCode, Animal] rule to be tested.
Implementing this would probably require the provision of outboard storage repositories typed to a fact type.
OutboardStorage#passivate(Collection<org.example.ZipCode>): null OutboardStorage#activate(): Collection<org.example.ZipCode> WorkingMemory#addStorageFor(Class, OutboardStorage): null
Using "Class" to match storage may not be optimal, what is really needed is a type matching pointcut as a single OutboardStorage type might be able to store many types of facts.
Just some ideas, I can only think about how to implement this in WM, but in theory something like this should be able to outperform operating system memory management and get past the 2GB barrier on intel for effective working memory size =)
