1 Software Developing: Domain Driven Design, looking at Persistence {anchor:Software Developing: Domain Driven Design, looking at Persistence} First I wrote {link:Software Developing: Domain Driven Design |url=http://www.pabrantes.net/blog/space/start/2007-01-14/1} where {link:jff|url=http://www.joaoferreira.org} asked a few general questions about DDD - since he isn't in the area - as a result I not only replied to his questions but also wrote a second post called {link:Software Developing: More About Domain Driven Design|url=http://www.pabrants.net/blog/space/start/2007-01-18/1} in order to make some concepts more clear. On the second time it was [jpmsi] that raised interesting questions regarding the subject, pointing out information tarpits for people that were new to DDD.\\ In this post I'll try to expand one of the subjects that [jpmsi] pointed out: Persistence in the DDD. __Note__: I won't be entering (yet) in the database structure, explaining how to do the Object-Relation bridge but that can be expect in a future post. Also, in the end of each item I'll be presenting some code, which, if javascript is enabled in the browser it will be collapsed, "example code" will be linkable and when clicked will show the code. To anyone who as read {link:Martin Fowler|url=http://www.martinfowler.com/}'s {link:Patterns of Enterprise Application Architecture|url=http://www.amazon.co.uk/o/ASIN/0321127420/ref=s9_asin_title_1/026-3265278-6114054} this post will seem very familiar. Since I'm talking about the book let me say it's a must read to anyone who is interested in software architectures. But let's start with the basics, and the easiest thing - yet still very important - which is defining what means persistence.\\ __Persistence__ is nothing more than the ability of preserving data beyond the execution of the program that has created it. Persistence is associated with some kind of database system, from a {link:RDBMS|url=http://en.wikipedia.org/wiki/RDBMS} to file system storage. When creating the architecture there are at least three things, regarding persistence, that should be in mind: * How to access the data layer: ** Mainly how {link:CRUD|url=http://en.wikipedia.org/wiki/Create%2C_read%2C_update_and_delete} operations are performed * What's the domain level behaviour regarding persistence ** How to persist every change done in a object that is in memory ** How to deal with objects relations ** Only load same object into memory once * Domain Objects must have some sort of relation with the data store in RDBMS ** Store in an object field the primary key for that object in the database Right now let's put the third point aside and assume that each domain object has an extra attribute which is the primary key for the object and it's ~~auto-magically~~ set. In the end I'll get there.\\ I'm sure there are various ways of solving such problems, but I'll be presenting - like I already said - {link:design patterns|url=http://en.wikipedia.org/wiki/Design_pattern_(computer_science)} that are based in Martin Fowler's previously mentioned book. \\ I'll present two different kinds of patterns, using Martin Fowler's names I'll be talking about the the so called __Data Source Architectural Patterns__ and the __Object-Relational Behavioural Pattern__. __Data Source Architectural Patterns__ These patterns are responsible for objects in the Data Layer that take care of the direct access to the data storage, abstracting the actual storage system to the rest of the application. Abstraction in this kind of situations is good, because with such abstraction the storage system can be replaced by another one and no other layer besides the Data Layer have to be re-written. __Pattern 1__: ~~Table Data Gateway~~ In this pattern exists an object that acts as a gateway to the RDBMS table - or tables depending of the database structure. That object has as it's interface the CRUD operations and any given domain object that needs to be persisted will be using this gateway. \\ Like the name implies the gateway is a gateway for a table, so there is one gateway object for each one of the tables that exist. As an example, let's imagine that we have a Book object which has a title and a number of pages - along with the primary key that has been previously mentioned. {collapsingBlock:title=Example code|visible=false} {code} public class Book { private static BookTableDataGateway gateway =new BookTableDataGateway(); private Integer id; private String title; private Integer numberOfPages; public Book(String title, Integer numberOfPages) { this.title = title; this.numberOfPages = numberOfPages; this.id = gateway.saveBook(title,numberOfPages); } public String getTitle() { return this.title; } public void setTitle(String title) { this.title = title; gateway.updateBook(this.id,this.title,this.numberOfPages); } public static Book readBook(Integer id) { return gateway.readBook(id); } // more code here } public class BookTableDataGateway { public Book createBook(String title, int numberOfPages) { // jdbc code here } public Book readBook(Integer id) { // jdbc code here } public void updateBook(Integer id, String title, int numberOfPages) { // jdbc code here } public void deleteBook(Integer id) { // jdbc code here } } {code} {collapsingBlock} Besides the problem of the maintenance overhead - since there's a new object for each table in the database - there is also the problem of how should the relations with other objects be solved: * using proxies? * also getting the other objects in memory? Each way as it's pros and cons so it's up to the person who implements to decide. ~~Side note: a simple way to test the domain layer when using this pattern is to replace gateways by {link:mock objects|url=http://en.wikipedia.org/wiki/Mock_object}.~~ __Pattern 2__: ~~Active Record~~ The Active Record is a very well known pattern, specially after the hype around {link:RoR|url=http://en.wikipedia.org/wiki/Ruby_on_Rails}, since it extensively uses Active Records.\\ In the Active Record pattern each domain object knows by itself how to deal with his persistence. This means that the data access logic is within the domain objects, in my opinion it makes the domain object dirty. This can bring problems along the development specially if the domain in question isn't trivial. But on the other hand if it's a simple domain a few conventions are used - like in RoR - then it can be a powerful, yet not very flexible, aide to the developer. {collapsingBlock:title=Example code|visible=false} {code} public class Book { private Integer id; private String title; private Integer numberOfPages; public Book(String title, Integer numberOfPages) { this.title = title; this.numberOfPages = numberOfPages; this.id = save(); } private void update() { // jdbc code to update the object } private Integer save() { //jdbc code to save the object } public void delete() { } public static Book readBookById(Integer Id) { } } {code} {collapsingBlock} __Pattern 3__: ~~Data Mapper~~ {image:img=DataMapper.png|align=float-right}In the two previous patterns the domain objects knew about the need to be persisted. Domain Objects would have in their code methods to access the data layer itself (Active record) or at delegate to another object which they knew the interface (table data gateway). In those two patterns there wasn't a real abstraction between the domain layer and the persistence layer.\\ This pattern is different, it introduces a true separation between both layers, the domain objects do __not__ know anything about the persistence. It brings a bit more complexity in the mapping system but in applications where developers want their domain to be truly independent from persistence a Data Mapper might be the solution. This pattern can be used along with the Lazy Loading Pattern - which will be described later on this post. Mainly what could happen is that when the load of a certain object is needed the load method of the mapper is called returning the object in question. __Object-Relational Behavioural Pattern__ The behavioural patterns have the objective of aiding the operations that are happening in the domain layer. By aiding I mean keep tracking of what's changing in the objects, bringing objects from and putting objects in the database. Each pattern presented here will solve a different problem, in complex systems all three can be found implemented. __Pattern 1__: ~~Unit of Work~~ This pattern introduces a way of keeping track of all in-memory modifications and make sure they'll get reproduced in the database. The idea is that inside a transaction flag all objects that has been modified or created an in the end of the transaction - in the __commit__ - find all flagged objects and write their modifications into the database. {collapsingBlock:title=Example code|visible=false} {code} public class UnitOfWork { List<DomainObject> newObjects; List<DomainObject> dirtyObjects; List<DomainObject> deletedObjects; public UnitOfWork() { this.newObjects = new ArrayList<DomainObject>(); this.dirtyObjects = new ArrayList<DomainObject>(); this.deletedObjects = new ArrayList<DomainObject>(); } public void markNewObject(DomainObject object) { newObjects.add(object); } public void markDirty(DomainObject object) { dirtyObjects.add(object); } public void markDeleted(DomainObject object) { deletedObjects.add(object); } public void commit() { boolean rollBack = false; TransactionalSystem.openTransaction(); try { for(DomainObject object : newObjects) { // code to create new object } // cycles for the other two lists }catch(PersistenceException exception) { rollBack = true; exception.printStackTrace(); } if(rollback) { rollBack(); } else { TransactionalSystem.commit(); } } public void rollBack() { TransactionalSystem.fail(); newObjects.removeAll(); dirtyObjects.removeAll(); deletedObjects.removeAll(); } } {code} {collapsingBlock} __Pattern 2__: ~~Identity Map~~ This pattern has two main objectives, avoid data incoherence and speed up the materialization of objects.\\ The Identity Map is nothing more than a cache, which only materializes objects from the database if they aren't already in memory, it's behaviour can be seen in the activity diagram present below. {image:img=IdentityMap.png|align=center} This pattern avoids data incoherences because when a object is in memory it only exists one instance of it, so even if somehow it's asked to be read from database it will be returned the same instance and at any time there will be two different instances of the same object making it able for the same object to have different internal states, which would be a severe data incoherence. {collapsingBlock:title=Example code|visible=false} {code} public class IdentityMap { private Map<Integer,DomainObject> objectsCache; private DatabaseConnector connector; public IdentityMap() { objectsCache = new HashMap<Integer,DomainObject); connector = new DatabaseConnector(); } public DomainObject getObjectWithId(Integer id) { DomainObject object = objectsCache.get(id); return (object!=null) object : connector.materializeObjectWithId(id); } } {code} {collapsingBlock} I don't know if it's obvious but what has been called Database Connector in these example can - and should be - one of the patterns that have already been described under the Data Source Patterns. __Pattern 3__: ~~Lazy Load~~ {image:img=LazyLoad.png|align=float-left}Lazy loading is the ability to load needed attributes only when requested. In Domain Driven Design this is extremely useful since domain objects are strongly related with each other. Mostly because if every attribute of a domain object would be resolved and materialized when a single object would be loaded the entire domain model could be loaded into memory. In a small model this behaviour might not be a problem but, in a big one it is. The idea is to use ~~something~~ that represents the object but it's not the object itself. The ~~something~~ mentioned is called a __proxy__, the lazy load pattern is often used with {link:Proxy|url=http://en.wikipedia.org/wiki/Proxy_pattern} pattern, more specific with the virtual proxy. I've said often because there are other ways of achieving lazy load besides proxying, such as value holders. As an example imagine that a Book as a relation with Authors, one book may have many Authors. Both Book and Author are domain objects. {collapsingBlock:title=Example code|visible=false} {code} public Author implements IAuthor { private String name; // ... public String getName() { return name; } } public AuthorProxy implements IAuthor{ private Integer authorsId; private Author author = null; public AuthorProxy(Integer idForRealObject) { this.authorsId = idForRealObject; } public String getName() { if(author==null) { author = connector.materializeAuthorWithId(authorsId); } return author.getName(); } } public class Book implements IBook { private Integer id; private String title; private Integer numberOfPages; Collection<IAuthor> authors; public Collection<IAuthor> getAuthors() { return authors; } } {code} {collapsingBlock} __Keeping track of what is what__ Like it has been said before the in-memory objects need to have some relation with the data stored in the RDBM, that relation is kept by a new field in the object which has the key for the object in the relational table. The question to ask is, where does that identification used as primary key come from? There are two different approaches: 1. It can be generated in the application 1. It can be generated by the RDBMS If the identification is generated by the application, it should be kept in mind the following: * The Identification numbers have to be unique, at least for each class and, depending how the domain is mapped in the relational database, it's hierarchy. * The generation and attribution of such identification needs to be transactional. There are persistence frameworks that uses this strategy, some implementing more than a way different way of calculating the identification numbers, thought there is a simple algorithm, called the {link: Hi/lo Algorithm|url=http://www.cs.uaf.edu/~cs301/notes/Chapter5/node5.html} that is often used due to it's efficiency. This algorithm generates the next identification based in some parameters.\\ A strategy is to keep those parameters in a table in database and use them to get the new object id. On the other hand if the choice is to let the RDBMS take care of the identification numbers, using for example AUTO_INCREMENT in the keys, there are also ways for retrieving the identification number.\\ It's usual for database driver connectors to provide a certain interface that allows the access to the generated row from the performed query. But, even if such functionality is not available there's always the SQL way to retrieve the maximum - so the latest - identification from a given table. In the last case should be kept in mind that the creating and read of the identification number should be done in the same transaction in order to be sure that it's the correct identification that is being read. __Conclusions__ It's true that already exists frameworks that deliver the behaviours described in this post and that can be - and probably should be- used instead of implementing all this from the scratch. An example of one of those frameworks is {link:hibernate|url=http://www.hibernate.org/}.\\ I do defend that if there are good frameworks they should be used instead of re-inventing the wheel. Though, the concepts that are underlying to such frameworks should be understood in order to gain the ability of analysis and to understand "how the show is running backstage" in case of some strange thing happens. After this post I hope some of those concepts and ideas were made clear. {tagometer:inline=true} 2007-03-20 19:13:20.674 2007-02-28 01:42:46.472 I have privately discussed this with my good friend Paulo, but I thought that I should make a comment here anyway. I believe the article is very well written. It approaches topics I think are very relevant in the context of DDD (the thinks that make it work). I even learned a couple of things myself :) __Why these topics?__ The first thing I should explain is why I think these are important topics. To a purist DDD developer, these subjects are pretty much non-existent. We believe the underlaying architecture takes care of that stuff to us. Rarely do we have to go down to the level of the inner workings of the frameworks. However, when I explain the merits and workings of DDD to some old school developer (read: people who only understand SQL+VB and the concept of a domain model is somewhat blurred with ER models) I get stuck to explaining them in terms they understand. And still they often end up asking "But wait... where do you code the services layer?". So, and even though Paulo as explained pretty well how do relations between objects work (keys and such), I now raise the question on how one can invoke these methods from a transactional perspective. With SOAs, the transactional unit is often a service, and its invocation generates a unique transactional context in which all changes occur (can you find out why simplistically speaking calling services within services is a bad idea?). How does DDD get around that (bear in mind that we strive for transparency from the perpective of business code)? I've seen a couple of perspectives (dare I say "aspects"?), but I urge anyone to come up with their own views :P __Yet another point__ Another point that I think is worth noting is object search. Another frequent question I get is how do I do a search with a set of criteria. Typically, developers create a database view, with fields for every criteria included in the search, plus the results columns intended. From there, a DAO is created to use with this view - with more or less flexibility concerning the use of criteria: use of conjunction (And, Or, ...), operators (Like, In, Is, Between, ...). This DAO will then yield something like a datatable (.NET specifics, sorry :P) which we'll use to present the results. How can we reach this flexibility with DDD? To this, I do not know many approaches, but I promise I'll look into the possibilities Hibernate offers. 2007-03-14 11:47:24.536 2007-03-14 11:47:24.39 {quote} With SOAs, the transactional unit is often a service, {quote} As you know - since you've already been developing code at FenixEdu - a thin service layer can be present in DDD approaches offering transactional support. By thin layer I mean ~~dumb services~~, in other words, services that are not aware of business logic - that will be taken care by the objects - and will only do gets and sets of objects. In my opinion the introduction of such thing layer might be a mixing between SOA and DDD. But it seems one of the best approaches to deal with transactional context, still why don't you share some of the ones you've already seen? But... now that you've mentioned aspects, that seems an interesting interesting approach! What's the main idea? Mark a setter as transactional? Though I guess that there would be a drawback the compilation time would increase drastically (remember when we tried aspects on Fenix for Software Testing class?). Though a selective weaving might fix that problem, not sure about that though. {quote} Another point that I think is worth noting is object search. Another frequent question I get is how do I do a search with a set of criteria. {quote} To me that's probably one of the biggest problems you might have in DDD, or maybe I just lack of experience on such problem. Using DAO's in DDD can be done, although that's punching DDD approach in the face. So I prefer to leave it aside.\\ Using OQL-like functionalities like the ones given by Hibernate - or any other persistence framework - can be an option, although I don't know if will be a really good one. Other idea - never tried it so I'll just hope this doesn't backfire on me - was to use {link:lucene|url=http://lucene.apache.org/} as your domain object indexer. With following "rules": 1. You wouldn't be indexing the object itself but only certain object fields. 1. Those fields would be marked as index on your (possible) Domain Modelling Language 1. There would have to exist a re-indexing system for creation, update and deletion of domain objects. I already have experience with lucene, {link:dspace|url=http://www.dspace.org/} uses it to search in repository's meta-information. Just to leave an idea of performance I can retrieve hundreds - if I'm not wrong the largest search result I got was something around 4000 hits - of relevant hits on a search in something like __one__ second. Other approaches to search might be: * Indexed reading where you still mark on your domain modelling language certain field as indexed and the DDD framework allows you to apply some sort of criteria reading. This is almost like OQL although not breaking the DDD concepts - at least I don't think it does. * In Memory Search, read all your search domain into memory and iterate through it finding the objects that do match your criteria. This is definitely __not__ a good option since it does not scale at all. Any other ideas are always welcomed. 2007-03-17 20:14:48.424 2007-03-17 20:14:48.093 {quote}As you know - since you've already been developing code at FenixEdu - a thin service layer can be present in DDD approaches offering transactional support. By thin layer I mean dumb services, in other words, services that are not aware of business logic - that will be taken care by the objects - and will only do gets and sets of objects.{quote} Well, I was just mentioning the standard approach developers often use with SOA. In my current project, our project leader decided to use a ThreadTransactionManagar that is used in every entry-point on the service layer (I although I do not want to elaborate further, I ca still say I dislike that approach). Basically the transaction manager offers one transaction per thread, allowing us to use nested service calls :S. {quote}But… now that you've mentioned aspects, that seems an interesting interesting approach! What's the main idea? Mark a setter as transactional?{quote} When I mentioned Aspects, I was referring to the process of adapting an already framework to transparently use the transactional system. I have had some time with NHibernate now, so I can safely say it is not a good think when programming DDD. First, there is the need to create a Session context for each service (transactional unit), and that does not merge well with our DDD approach. After that, for each modification we have to call object.save() to persist changes. And to create/delete we have to call static methods on the Domain class. This is not a very good DDD way of doing things. My naïve Aspects comment was about calling save after each setter (which in the end does not offer a full solution)... Comments are more than welcome... I did thought about some other approaches, namely only persisting changes on the persistence layer upon commiting, and walking the graph of objects to determine creations/updates/deletes. However this approach requires the use of an Software Transaction Memory manager, which is exactly what you guys use @ fénix. {quote}Other idea - never tried it so I'll just hope this doesn't backfire on me - was to use >>lucene as your domain object indexer. With following "rules":{quote} Regarding searches, I will add only this: you are still throwing away performance when you want to search across multiple tables (read joins). Should anyone come up with a good solution, please do tell me... Although I pretty much doubt such would exist... Edit: I found some good reading about the subject - http://debasishg.blogspot.com/2007/02/domain-driven-design-inject.html - http://debasishg.blogspot.com/2007/02/domain-driven-design-use-orm-backed.html And I would recommend reading some other posts that are off-topic for this discussion. 2007-03-19 10:28:25.022 2007-03-19 09:55:07.635 {quote} \[...] Basically the transaction manager offers one transaction per thread, allowing us to use nested service calls :S. {quote} Nested transactions sometimes can be good, although it's something that should be used carefully. {quote} My naïve Aspects comment was about calling save after each setter (which in the end does not offer a full solution)… Comments are more than welcome... {quote} What about using aspects with a Unit of Work? The aspects would simply mark an object as new, dirty or delete on the unit of work. Still a Transaction Manager would be needed to then collect the data from the unit of work and persist such changes... (No coments, back to square zero). {quote} Regarding searches, I will add only this: you are still throwing away performance when you want to search across multiple tables (read joins). {quote} Yes, even if everything is loaded into memory and you don't need to worry about the materialization time a search in memory cannot be compared with an optimized search of a RDBMS. Specially if you need to do navigation through the domain (read joins). There aren't perfect solutions, only tradeoffs. But it would be very (very!) nice to have a good solution for the problem. By the way, interesting links the ones you gave! 2007-03-20 01:27:43.792 2007-03-20 01:26:33.118 {quote}Nested transactions sometimes can be good, although it's something that should be used carefully.{quote} Notice how I did not say nested transactions, but nested service calls... On transaction per thread says it all :P {quote}(No coments, back to square zero).{quote} Back indeed. I still think only a SMT will solve this... Regarding searches, I've talked to Mr. Gil, and he also thinks that having a domain representation over a view, in order to materialize searches faster is a pretty good trade-off. Kind of having a Search<DomainObject> concept in your domain... What say you? 2007-03-20 13:39:19.29 2007-03-20 10:40:37.28 {quote} Notice how I did not say nested transactions, but nested service calls… On transaction per thread says it all :P {quote} Hmmm, indeed I misunderstood it, my apologies.\\ I was thinking that each service call would be a thread and the transactions representing those threads would be subtransactions of the "top-level" transaction. {quote} Regarding searches, I've talked to Mr. Gil, and he also thinks that having a domain representation over a view, in order to materialize searches faster is a pretty good trade-off. Kind of having a Search<DomainObject> concept in your domain… What say you? {quote} So what you are saying is that the Search object would be nothing more than a view that could allow you to search objects through criterias, right? Well to me it seems a good trade-off, although I think it's dangerously approaching a DAO pattern (a generic DAO, but yet a DAO). When I was thinking about using Lucene my idea was similar to that one, but like I was expecting such idea did backfire on me. Mr. Luis Cruz (liked Mr.) pointed me out that Lucene isn't transactional aware and we would have consistency problems when a transaction would abort. And making the transaction index aware is..hmmm...spooky? Anyway I'll leave lucene idea for now. 2007-03-20 11:07:20.363 2007-03-20 11:02:12.019 {quote}...I think it's dangerously approaching a DAO pattern (a generic DAO, but yet a DAO).{quote} Well, I do think that it should observe a DAO behaviour, since the DAOs used with persistence APIs have all the query facilities we will need to use (query languages, like OQL and the likes). Maybe it's a good idea not to mix these objects with our domain, and let them be a specialized construction to deal with a specialized problem (searches). 2007-03-20 11:17:17.121 2007-03-20 11:17:17.016 {quote} Maybe it's a good idea not to mix these objects with our domain, and let them be a specialized construction to deal with a specialized problem (searches). {quote} Well yes, if you do pay attention and do not mix DAOs on the domain itself I think I can agree with such solution. You could abstract the DAOs and provide a search interface or api that would allow to use DAOs indirectly. 2007-03-20 17:50:00.417 2007-03-20 17:50:00.293 I think we are approaching two distinct, yet closely related, subjects: 1. The use of DDD 1. Persistence of an OO model This distinction is important to clarify things as we can have the first without the second (bare with me) and the second without the first. We can also have the first with an OO DB which changes many of the assumptions made. Now, considering searches over the graph of objects composing the OO model resulting of a DDD. I don't think the ~~Search<DomainObject> concept~~ that João described above was completly understood. The suggestion was to consider a view in a database as a regular table. So, if your persistence framework maps objects to tables, it can map and object into a view. If you then ask the framework to provide all objects of that type (a readAll) it will provide objects representing the result of a complex query "transparently". A simple example: a bank. Lets assume that the core of the bank's domain is the relation between accounts and their owners. But "Customer" is also a relevant concept for us (the bank) and we want to decide between those customers that we would rather loose and those customers we want to keep. {image:img=m4ktub/ddd-example-1.png|align=center} I've just invented a stereotype in the example to mark those domain concepts that we will map into a view. But I believe that they are domain concepts with the same relevance as the others. They just have different domain rules. For example "a TerribleCustomer can't open any more accounts" o "a PremiumCustomer has +1% in the interest rate". The fact that these domain classes can be implemented in the database as two views it's problably only an optimization. But this isn't the main problem enounced here. {quote}The traversal of the object graph corresponding to the application's domain is more inefficient than direct query in a database and materialization of the result set over a database.{quote} Well, I believe that is more inefficient if you are doing a complex (several levels deep) search, most of the graph is not in memory, and you need to materialize those intermediate objects to do the traversal. If the intended query is just 1 or two levels deep then processor speed higly overcomes the network latency/traffic normally associated with the communication to a database. Offcourse I have to do some tests to back this up (all those Prevayler related tests are out of date and unrealistic). 2007-03-20 20:27:44.144 2007-03-20 20:25:39.936 {quote}I don't think the Search<DomainObject> concept that João described above was completly understood. The suggestion was to consider a view in a database as a regular table. So, if your persistence framework maps objects to tables, it can map and object into a view. If you then ask the framework to provide all objects of that type (a readAll) it will provide objects representing the result of a complex query "transparently".{quote} Well, to me the main problem with this approach is the materialization of the objects resulting from a search, plus all the related objects needed to present the results, when such materialization was (probably) already made by the database engine. I'll present the same problem as I did to you personally yesterday: Consider a costumer with orders. You wish to list all costumers plus it's most recent order. You would create an index on the date of the order, and join the costumer table with the order table on the most recent order. The query will discard quickly the orders we don't want because it simply transverses the BTree to find our order. However, when done in memory, upon materializing the results, you'll have to find the most recent order by iterating manually through the orders of the costumer, materializing them all to do the comparison and find the most recent. This is a very simple example that (I hope) can demonstrate my point. 2007-03-22 15:55:36.654 2007-03-21 10:58:33.16 First of all my apologies for a late reply... Now back to our interesting discussion, {quote:m4ktub} I don't think the Search<DomainObject> concept that João described above was completly understood. The suggestion was to consider a view in a database as a regular table. So, if your persistence framework maps objects to tables, it can map and object into a view. If you then ask the framework to provide all objects of that type (a readAll) it will provide objects representing the result of a complex query "transparently". {quote} Yes, I hadn't understand like that, thanks for pointing it out.\\ The idea itself seems good and more efficient, afterall database engines do that at it's best. But you still defend that for one or two levels and having all the domain materialized the search in memory will be faster...hmm... I'm not really sure about that, but neither am I sure about the other way around! When can we see your tests result data? ;-) {quote:jmpsi} (...) when such materialization was (probably) already made by the database engine. {quote} I'm sorry I didn't quite follow you here. The database already had made the materialization? We're talking about relational databases there's no concept of materializing an object. I must have misunderstood something about what you were saying. Could you please clarify? {quote:jmpsi} However, when done in memory, upon materializing the results, you'll have to find the most recent order by iterating manually through the orders of the costumer, materializing them all to do the comparison and find the most recent. This is a very simple example that (I hope) can demonstrate my point. {quote} That is not completely true - let me be a bit of a troll - imagine for example that you keep an ordered list of orders, using this implementation you could simply do a direct access to the first or last item of the list (depending on how you were ordering it).\\ But yes, I did understand what you were saying. Doing a readAll and iterating the resulting list in order to find which objects are matching our criteria is, may I dare to say always, a nasty implementation that won't scale properly. 2007-03-23 01:38:31.406 2007-03-23 01:34:24.036 {quote}I'm sorry I didn't quite follow you here. The database already had made the materialization? We're talking about relational databases there's no concept of materializing an object. I must have misunderstood something about what you were saying. Could you please clarify?{quote} When I talked about this kind of materialization I was talking about the DB engine following relations (joining) and applying restrictions. This is not to be mistaken with the materialization of objects. {quote}That is not completely true - let me be a bit of a troll - imagine for example that you keep an ordered list of orders, using this implementation you could simply do a direct access to the first or last item of the list (depending on how you were ordering it).{quote} Well, introduce me to the persistence framework that does that and I'll take a byte at it. Still, you are still materializing all these objects into memory (because the framework now has the burden of sorting them) to use just one. Even if the sorting is delegated upon the DB engine, you would still materialize an whole set of proxies (at best) to use only just one. Multiply that with your entire result set and there you have it... Anyway, allow me to provide with yet another example... Say you have a class with a state machine. This class has states s1, .., sn. Now you wish to provide a way for the user to search through your 1e7 records with a criteria that maps to a set of check boxes representing each state. Selecting none searches objects with any state. Selecting some represents the union of states selected. You can simply add an index on the state, and using the IN operand you get some pretty quick results (the engine is quick to discard unselected states...). Would doing that in memory be anywhere near that efficient?. And yet another question: say you wish to do paging on the results... Using a parametrized query on the view can quickly discard unwanted results... Doing that in memory can be disgraceful, even only materializing a list of, say, 1000 proxies to use only 20. My point being, the sum of inefficiencies DDD presents in this field can very well be the deciding factor not to use it in a performance critical environment... 2007-03-23 12:21:05.176 2007-03-23 10:35:36.38 {quote}Anyway, allow me to provide with yet another example… Say you have a class with a state machine. This class has states s1, .., sn. Now you wish to provide a way for the user to search through your 1e7 records with a criteria that maps to a set of check boxes representing each state. Selecting none searches objects with any state. Selecting some represents the union of states selected. - __jpmsi__{quote} And if a tell you that we keep an hashtable/structure with the mapping state->{elements in that state}. And that we identify that structure as and important concept in our domain. And for efficiency reasons that structure is mapped into a view like the examples above. In many cases we can transform a problem to match a pattern more familiar to DDD. Nevertheless the paging issue is terrible. I currently don't know a solution that does not require all objects to be in memory. Paging really requires a deep integration with the presistency engine. {quote}When can we see your tests result data? ;-) - __pabrantes__{quote} It's coming up slow. But it's coming up more like a curiosity and a dismissal (or confirmation) of some dogmas than a definitive answer to most of the questions we've made here. Sometime soon ... 2007-03-23 12:36:17.828 2007-03-23 12:36:17.699 {quote:m4ktub} Nevertheless the paging issue is terrible. I currently don't know a solution that does not require all objects to be in memory. Paging really requires a deep integration with the presistency engine. {quote} Well if you don't mind having some nasty hacks under the hood your persistence engine could materialize an object, let's call it Pager, that would do the job. The code would be something like (keeping in mind that some verifications should be done and are not present in the example): {code} public class Pager<E> { Integer pageSize; List<Integer> idsToMaterialize; public Pager(Integer pageSize, List<Integer> idsToMaterialize) { this.pageSize = pageSize; this.idsToMaterialize = idsToMaterialize; } public List<E> getPage(Integer page) { List<Integer> pageIds = idsToMaterialize.subList(this.pageSize*page,this.pageSize*page+pageSize); List<E> elements = new ArrayList<E>(); for(Integer id : pageIds) { elements.add(PersistenceEngine.materialize(id)); } return elements; } public int getTotalElements() { return idsToMaterialize.size(); } } {code} This solution is based on how dspace perform searches and how I page them. As I previously mentioned dspace uses Lucene and lucene returns an object that among other things contains the total number of elements matching our search and a list of a certain identifier that allows us to know which object is he talking about. There's one thing bugging me though, I would like that __materialize__ method would also receive the class of the object. But I can't say E.class, although I could say: {code} E x = new E(); Class realClass = x.getClass(); {code} But I think it's...err...too dirty! Any better ideas about this? \\ With this system you could request the persistent system to return a certain list as a Pager instead of a List. But no order on the list, which in many cases might be a problem. 2007-03-24 23:34:50.156 2007-03-24 23:34:04.8 {quote}And if a tell you that we keep an hashtable/structure with the mapping state->{elements in that state}. And that we identify that structure as and important concept in our domain. And for efficiency reasons that structure is mapped into a view like the examples above. In many cases we can transform a problem to match a pattern more familiar to DDD.{quote} But this a different approach to my stance in DDD. I believe that in terms of the Domain of the application, this concept is not relevant at all.. It stands as a solution to a problem that arises from our choices of the programming architecture/methodologies. This part should be of no concern to the domain logic, and I currently think that it should be presented as an infrastructural service to the application. Edit: I think it stands more clearly if I'd ask you to explain to a client the need of a business representation of a search concept. In terms of his business, it is meaningless, therefor it does not belong to the domain. 2007-03-27 12:23:53.49 2007-03-26 11:04:43.077 {quote}Well if you don't mind having some nasty hacks under the hood your persistence engine could materialize an object, let's call it Pager, that would do the job. The code would be something like (keeping in mind that some verifications should be done and are not present in the example):{quote} I reply the same as I did to Mr. Gil. I really think this approach is valid, but not a part of DDD. It is a solution to a specific problem that does not concern business logic. And how about (new E()).getClass(); :D:D:D 2007-03-26 11:07:21.728 2007-03-26 11:07:21.523 I agree with you. Most of these problems are infrastructural. There should be direct support for this situations, like paging, in the persistence framework we are using. All other solutions are hacks around it and often are not relevant concepts in the application's domain. __just thinking out loud , pay not attention__: Off course the real challenge is to provide a paging solution that is well integrated in the model. Ideally we would be able to do a query over the domain, with some sort of OQL, and have paging in an efficient way without ever leaving the comfort of the OO model. Something in the lines of {code} QueryPager<Item> pager = Query.pagedExecute(Item.class, "orders[date > %1].customer.country == 'PT'", lastWeek); print(pager.getNumberOfPages(pageSize)); for (Page page : pager.getPages(pageSize)) { for (Item item : page.getElements()) { for (Order order : (Collection<Order>) page.getElements(item)) { Customer customer = (Customer) page.getElement(item, order); assert(page.getElement(item, order, customer) == "PT"); } } } {code} I have to check some existing OQL solutions and how they try to optimize the query. (more one thing to do :-) ) {quote}And how about (new E()).getClass(){quote} E, in the example, is not an example class, it's a generic type. So you can't even do new E(). 2007-03-27 15:14:34.164 2007-03-27 14:36:11.762 {quote:m4tub} I agree with you. Most of these problems are infrastructural. {quote} Yes, I also do agree with that. {quote:m4ktub} There should be direct support for this situations, like paging, in the persistence framework we are using. {quote} Does this really makes sense? Shouldn't be the persistence framework a small set of operations and then you would create more complex operations - such as paging - around it? {quote:m4ktub} E, in the example, is not an example class, it's a generic type. So you can't even do new E(). {quote} Yes, you're totally right, it seems my brain froze while writting that part. After all generics are compile time, not run time. If on runtime there's no generics you cannot make a reference to it's class. Oh well... It ~~seemed~~ a good idea to read objects using the class. 2007-03-27 23:21:06.444 2007-03-27 23:21:06.262 {quote:m4ktub}Something in the lines of{quote} Woa... I think you may be getting near to the level of .Net's LINQ with that OQL query (OQL is not and should not be new to me, but I just could no resist throwing that one). Read {link:.Net's LINQ|http://msdn2.microsoft.com/en-us/library/ms364068(vs.80).aspx} for some examples (thanks Mr. ~~<censored>~~ for that). From what I read, you created a collection of pages (as in page 1, records 1-10, page 2, records 11-20 and so on). Then you materialize the elements by iterating the page elements? Am I understanding correctly? If so, wouldn't it be better to have a {code} Query.pagedExecute(Class c, String query, int begin, int offset, Object... params); {code} {quote:pabrantes}Does this really makes sense? Shouldn't be the persistence framework a small set of operations and then you would create more complex operations - such as paging - around it?{quote} My € 0,02 is that paging is something you will be wanting to do at database level... 2007-03-29 08:40:42.121 2007-03-28 09:32:52.672 {quote}Does this really makes sense? Shouldn't be the persistence framework a small set of operations and then you would create more complex operations - such as paging - around it? __pabrantes__{quote} As we saw, the problem with that is the lack of control over the specific features of the DB that allow to improve performance. From the LINQ documentation: ~~The Standard Query Operators is an API that enables querying of any .NET array or collection.~~ I have only read a couple of lines but it looks like it's an in-memory query infrastructure that offers a language that is resembles SQL. So that does not solve our problem of a "transparently persisted OO model capable of offering efficient query with paging operations". I give a further look at Regarding the previous example, jpmsi is right, those parameters in the pagedExecute method make sense but my goal was other. I was focusing on the problem stated before where, after the query, you would need to do extra computation in memory and redo part of the filtering already made in the DB. So the example shows a Page structure that would give you for each page a list of Item, for each Item the list of orders that matched the query for that item, for each order, the corresponding customer, and for each customer it's country. This information would be obtained with one query, possibly directly from the DB, and stored efficiently (uh, the magic word, but I was only thinking on lazy loading :-D) in the Page structure thus preventing us to reiterate over the Item's orders. 2007-03-28 11:40:23.5 2007-03-28 11:40:23.357 Ok, finally I could obtain some of the results I've been promissing in this discussion. Note that these results are obtained without any formal method and the only way you can be sure is by doing the tests by youselves :-). {image:img=m4ktub/ddd-bench-a.png|align=center} As you can see, I've measured run-time of 3 strategies so higher is worst. I will let the conclusions for each one but I just want to focus the relation between __A__ and __C__ and what I will probably try to measure next. __A__ can be considered the worst case scenario of __C__ (oh, __C__ is the actual time measured, __C/100__ being the time of a single iteration). In fact I've also measured the time of the first iteration in __C__ and it was bery similar to the times of __A__. So we have to consider how many misses from memory take __C/100__ to a value higher than __B__. I've tryed to measure the time of loading a single object from the db but it was late in the night and the result had a lot of noise. Offcourse that if one deployment of the application used the __C__ strategy it would required much more memory and would need to mantain the consistency of all objects in memory over all the machines in a cluster. That would increase the time of __C__ and I'm not counting that here. A deployment like this would probably do fine with a single DB for a long time. The load would be surely in the frontend servers. In __B__ I believe that the load would be in the DB so scalability could be obtained through clustering of the DB. 2007-04-03 12:35:57.735 2007-04-03 12:35:57.593 It's unbelievable the time that __C/100__ took, smoking fast! I think we can say that this test kinda dismistifies the idea that searches on the DB are faster. Although in most of the applications will never have all the domain in memory. \\I guess one of the things you'll be trying to measure - you didn't mentioned what you were gonna try next - will be where between A and C situations memory searches are still faster than a view in the database (B). Also I think it would be interesting to know the time it takes to load an object, maybe one day that you won't be doing such tests late in the night? ;) I think we all should thank m4ktub for the valuable information he has posted! Thanks Claúdio :) 2007-04-03 22:43:36.154 2007-04-03 22:43:36.008 Don't get overexcited because I've finished the tests I wanted to do. The extra tests I've made include paging and the computation of a value with some cuts like João suggested. I've also redone B test because I've introduced a few more optimizations in the DB level (some extra indexes). {image:img=m4ktub/ddd-bench-b.png|align=center} As you can see the test from B was reduced to less than 1/3 of the original value. The __D__ test was ommited. In it I tryed to measure the time it takes to load a single record. It proved meaningless. It doesn't take long (around 0.30) and the "real" cases hardly ever loaded a single record. We normally loaded an entire 1-n relation. The __E__ test was a query like "SELECT SUM(...) ... WHERE status = <value> and date BETWEEN <begin> and <end>". The in memory computation required us to process each Order and filter the order by status and date and then compute the sum for all OrderItems in the orders that weren't filtered. As you can see, DB clearly beated the in-memory processing. João was righ when considering that database cut's based on indexes are very efficient compared to in-memory graph traveling. As an additional note I've found that, in __F__, most of the time was spent comparing dates (creating a calendar, seting the date and using compareTo()). The last group of tests paging. In-memory paging, that is __H__, includes creating a new List from the list of persisted objects, sorting that list and geting the subpage (subList(start, start+offset)). So, in-memory computation can beat the db in some situations. We lack the kind of infrastructure the database has when regarding query optimization, indexes, cuts, etc. That infrastructure could be partially implemented in a framework to optimize the selection of objects in memory and increase the cases were in-memory beasts communicating with the database. Nevertheless there is a huge effort in the DB backend and I'm not sure how much of that effort would be needed to replicate in such framework. We also have the scalability isssue ... :-) Guess this didn't gave us THE answer to our problems but at least we have some "pretty" graphics. Uh! Shinnny!!!. 2007-04-09 14:27:26.696 2007-04-09 12:19:01.254 {quote} Don't get overexcited\\ m4tub {quote} Damn...ok! {quote} As you can see the test from B was reduced to less than 1/3 of the original value.\\m4ktub {quote} First conclusion of the night: database optimizations __do__ work. :-P {quote} The E test was a query like (...) As an additional note I've found that, in F, most of the time was spent comparing dates (creating a calendar, seting the date and using compareTo()).\\ m4ktub {quote} Not that I'm being stuborn or any sort of evangelist but, such results might be due to the lack of optimization in the in-memory search code. There can be better search strategies rather then going from the all list and using compareTo(). Still, on that test the query kicked asses. {quote} The last group of tests paging. In-memory paging, that is H, includes creating a new List from the list of persisted objects, sorting that list and geting the subpage (subList(start, start+offset)).\\ m4ktub {quote} Too bad you have to do a copy...otherwise it would be smoking fast. You're making the copy because you're ordering the list, right? {quote} So, in-memory computation can beat the db in some situations. We lack the kind of infrastructure the database has when regarding query optimization, indexes, cuts, etc. That infrastructure could be partially implemented in a framework to optimize the selection of objects in memory\\ m4ktub {quote} So mainly what you are suggesting, is to bring a DB engine into our application? I'm not very sure about that, database engines are fine tuned and mature piece of software, a new framework would ~~probably~~ need a huge effort to achieve similar ideas. I say probably because I have no expertises in the area and it's just my common sense. Hence, I think we should leave database related subjects within the database. {quote} :-) Guess this didn't gave us THE answer to our problems but at least we have some "pretty" graphics. Uh! Shinnny!!!.\\ m4ktub {quote} Well there's no "THE" answer, but indeed we have some "Uh! Shinnny!!!" layout breaking - if I may add - graphics. Now for real, they may not give the ultimate answers, but still they give us a good insight. Now let's wait for João's feedback saying "AH! I told you" ~~laugh~~ 2007-04-10 00:45:13.543 2007-04-10 00:45:13.371 Wed Feb 28 01:43:34 WET 2007 Wed Feb 28 01:43:16 WET 2007 Wed Feb 28 01:43:24 WET 2007