The story about xxxProviderBaseCore.Fill and load/refresh

This is a long post but I would really appreciate any response to my questions.

                                    

We encountered an issue when trying to refresh entities from the database. We use deep loading to get the stuff into memory. We have to clients running in parallel and do changes on one of the clients. The changes are saved and then it was turn to hit the refresh button in the other client to load the changes in the other client. Now to the programmers surprise it turns out nothing happened.

-         What the … (programmer).

-         Easy now (that’s me).

 

After a little investigation we found the following.

 

To avoid the nuisance of having the same row in the database being represented by several different objects in memory (that sucks if you try to do a DeepSave and also worry about detecting and handling concurrent changes) entity tracking was turned on. Great, we get the same object each time we see that row (good if doing DeepSave no conflict with yourself anymore).

 

Now what was not so good was that when again calling DeepLoad or GetByPrimaryKey for an already loaded entity it turns out that for example GetByPrimaryKey does the expected thing. It constructs a reader with the latest information from the database. That reader is then passed to the xxxProviderBaseCore.Fill method that is supposed to turn that reader into a TList<xxx> of xxx instances with information from the database. Now this works if we havn’t seen the entity before but in xxxProviderBaseCore.Fill there is a conditional

 

if (!DataRepository.Provider.EnableEntityTracking || c.EntityState == EntityState.Added)

{

c.SuppressEntityEvents = true;

           c.ExerciseID = (System.Guid)reader["ExerciseID"];

           c.OriginalExerciseID = c.ExerciseID;

c.DbTimeStamp = (System.Byte[])reader["DbTimeStamp"];

           c.EntityTrackingKey = key;

           c.AcceptChanges();                                              

           c.SuppressEntityEvents = false;

}

rows.Add(c);

 

that determines whether the information is transferred from the reader onto the object. Now as you can see this will, when entity tracking is enabled, only happen when the object has EntityState.Added which is not the case when you already loaded the object once.

 

Now, I fail to see why we wouldn’t want to transfer that new fresh information from the database to the object. Is it because we don’t want a load to affect a shared object (I’m currently doing the load in my little part of the process but there are another bunch of modules in this process also having loaded the same object)? Or is the reason performance, i.e. we would spend a lot of time in the DeepLoad scenario trying to set values that already have been set.

 

Now, there is the xxxProviderBaseCore.RefreshEntity that does exactly what I want, given a reader or a DataSet. I see two possible solutions. Either xxxProviderBaseCore.Fill is allowed to transfer the values read from the database to the object no matter what (i.e. get rid of the “if” but keep the body of it). Or hack a corresponding DeepRefresh framework to get our object graph refreshed from the database.

Now if someone just could enlighten me as to where I’m thinking wrong when wanting to loose the if it might be easier to accept the difference in amount of work implementing the Refresh code (that I feel is currently lacking).

 

Or the last desperate resort for a quick fix would be to let go of all references to everything you’ve loaded (since weak referencing is used by the Locator in the ObjectBuilder, flush the cache (if it is enabled) and force a garbage collection (wohaaa ….) Now that should hopefully get rid of all references to objects in the object graph letting us once more load the stuff back in with fresh values. (Just the thought of this gives me the creeps).

 

Thanks for bearing with me this far. Does anyone have any suggestions or comments?

 

 

 

Well, the thing is, when an object is saved, it's removed from the locator altogether. So then when you're filling the entitiy again, it should be an added entity again and not be filled.  At least that's the intent.  But if you're certian of the behavior then we can change it.

Nope, now I get why you track entities. It is to make sure that you don't ruin any changes that has been done since you loaded an entity. I've implemented a property CurrentLoadPolicy on the provider to control how Load behaves with respect to changed entities. I'll post it in contributions when I've done some more tests. If you like it feel free to incorporate it. It's a non-breaking change.

What is the desired behavior when it comes to loading entities while entity tracking is enabled?

This is how it works right now. If you load some entities while entity tracking is enabled the tracker will give you the same instance back each time a specific row is read from the database. This is good if you (like me) have some sort of graph that you are working on. I most certainly do not want the same entity being represented by several instances.

Now, what really confused me is what happens when things start to change. As the tracking works today it has some strange effects (at least in my opinion). First it will prevent changes in the database from being loaded. If you load an entity e1 and there is a change of e1 in the database you will not be able to load that new version of e1 UNLESS you first modify it. Modified objects are not tracked and therefore a fresh instance will be created for the modified object when it is read from the database. This leaves us with the possibility of having the same instance floating around both in a new version (the changes loaded from the DB) and the modified version.

I have the need to load changes (i.e. refresh entities from the DB) even if I have not modified an entity. What I've (at least partially) done is the following. I've added a property DataRepository.Provier.CurrentLoadPolicy. This property can different values (currently) LoadNew, PreserveChanges, and DiscardChanges.

The thought of LoadNew was to preserve the behavior I thought was the default behavior today, i.e. only load new entities, not to refresh any entities already loaded. Now this is clearly not the way it works. But that was what I believed until recently.

PreserveChanges would refresh any unchanged entities with values from the database.

DiscardChanges would refresh both unchanged and changed entities with value from the database. Changed entities would become unchanged after a load with LoadPolicy.DiscardChanges.

I still need to do something, so my question is, is there anyone out there that have any wishes or thoughts about how this should work? Or are you happy with it as it works today?

Should entity tracking always return the same instance no matter it being modified or not? Is there a time when an entity no longer should be tracked for example when it is saved? Or should it keep on being tracked until the last reference to it is released? If so we would be certain that an entity always was represented by one and only one instance (unless we turned entity tracking of, cloned the entity, or otherwise actively created a second instance). If entities are always tracked and we wanted to compare a changed entity with what's in the database then you would have to clone the changed instance for the entity and then do a load with LoadPolicy.DiscardChanges to get the original updated with values from the database. You could then compare the two instances.

 

 

 

Magnus,

that's a very interesting scenario, it sounds like something I may run into as well in my application.  For numerous reasons we have chosen to go with an optimistic locking strategy, using a TimeStamp field (we call it "version") on just about every entity table that really matters, which .netTiers handles beautifully.  This however sounds like it would be a great help for this situation.  The check in the Fill() method can be slightly tweaked to check if there is any difference in the timestamp field along with the other checks, and if so, it should refresh the values in the currently loaded entity from the already acquired DataReader.

if (!DataRepository.Provider.EnableEntityTracking || c.EntityState == EntityState.Added)

can be changed to

byte[] dbVersion = reader["version"];

if (!DataRepository.Provider.EnableEntityTracking || c.EntityState == EntityState.Added || c.IsDifferentVersion(dbVersion))

This IsDifferentVersion() method could either be handled by .netTiers automatically, it already knows which entities are timestamped, or just create it as a vritual method on the base entity returning false, and it would be the reponsibility of the end programmer to tweak it for their own purposes.

Obviously this would require an extra TimeStamp field in each of the pertinent entity tables, but I think that is a small price to pay, and you also get the bonus of detecting changes on calling any of the Save() methods, .netTiers handles this automatically, detecting if the timestamps are different and throws a ConcurrencyException when this happens.

This looks to me like a workable solution, try to shoot any holes in it.  I hope it helps you out, and let us know which direction you're going in because I have a feeling this is something many of us will encounter eventually.
 

Thanks for showing some interest!

My first thought was similar to yours. However, I want to avoid breaking changes in netTiers unless absolutely necessary. There are a lot of people out there that doesn't use TimeStamping (I do thou). I've currently tweaked netTiers a bit for our own purposes. We are currently running code looking like this.

if (!DataRepository.Provider.EnableEntityTracking ||
    c.EntityState == EntityState.Added ||
    (DataRepository.Provider.EnableEntityTracking &&
        ((DataRepository.Provider.CurrentLoadPolicy == LoadPolicy.PreserveChanges && c.EntityState == EntityState.Unchanged) ||
         (DataRepository.Provider.CurrentLoadPolicy == LoadPolicy.DiscardChanges && (c.EntityState == EntityState.Unchanged ||
                                                                                                                               c.EntityState == EntityState.Changed)))))

This seems to me to be a more general principle that is applicable even if you are not using TimeStamping. The question is what is the expected when using EntityTracking and you start to modify loaded entities. The key issue is really when you change the primary key value for an loaded entity to something else. But I do believe that the most intuitive behaviour is to get the modified entity returned to you when loading unless you've specified DiscardChanges. This however means that I'll have to alter the tracking behaviour so modified entities are tracked as well. So the question is how many of you out there depend on the somewhat unintuitive behaviour of today? Will this be a major issue? If so I guess I'll have to go for a switch at generation time where you specify if you want todays behaviour or the new improved one, but I rather not.

Unless I hear any objections that'll be what I'll implement then it remains to be seen if the netTiers team deems it useful enough to incorporate it. I'll post the changes in Contributions so you can grab it from there if you are interested.