|
|
Comments and Discussions
|
|
 |
|
Thanks!
The problem is probably in the RebuildFromScratch method, the storage file has an entry for a deleted guid and this method should handle that correctly.
Its the man, not the machine - Chuck Yeager
If at first you don't succeed... get a better publicist
If the final destination is death, then we should enjoy every second of the journey.
|
|
|
|
|
Thanks, I'd found that part of the code since I posted. Stepping through the code it appears that the RebuildFromScratch() is incorrectly identifying these objects as not having been deleted.
private bool isDeleted(byte[] hdr)
{
return (hdr[(int)HDR_POS.Flags] & (byte)1) > 0;
}
Is returning false, for entities that have been deleted. The value of hdr is {77, 71, 82, 4, 0, 0, 0, 0, 0, 0, 0, 0, 19, 160, 1, 0, 0, 0} So the 16th value(which stores the flag) is 0, which means it's not deleted
Also while rebuilding the view there are a number of entities with an all zero GUID which isDeleted is returning true for.
I'm able to reproduce this even after deleting the database and starting again by just creating 1 or 2 objects in the database, deleting them, then removing the "Views" folder and re-starting my app.
Any ideas?
|
|
|
|
|
Can you please debug the StorageFile.WriteData method and see if it is setting the flag to 1 :
if (deleted)
hdr[(int)HDR_POS.Flags] = (byte)1;
Its the man, not the machine - Chuck Yeager
If at first you don't succeed... get a better publicist
If the final destination is death, then we should enjoy every second of the journey.
|
|
|
|
|
Yeah, the flag is being set to 1 initially in StorageFile.WriteData so it looks like it's an issue with rebuilding the views.
The only objects that are being read back from the database which are marked as deleted are the objects I mentioned before where the docid is all zeros.
|
|
|
|
|
The code seems logically ok, the only thing that I can think of is when deleting, the index for the guid is not up to date so it can't find the the guid to mark them as deleted in :
private void DeleteRowsWith(Guid guid)
{
WAHBitArray gc = QueryColumnExpression(_docid, RDBExpression.Equal, guid);
_deletedRows.InPlaceOR(gc);
}
Its the man, not the machine - Chuck Yeager
If at first you don't succeed... get a better publicist
If the final destination is death, then we should enjoy every second of the journey.
|
|
|
|
|
I'm just trying to create a demo of this bug. At the moment it's wrapped up application code so I'm just seeing if I can separate it out into a simple test case. I'll get back to you later on it.
Thanks for the help.
|
|
|
|
|
So I've created a simple test case for the bug I'm seeing.
- I create a new DB
- populate it
- delete 1 object
- close the connection to the DB so I can remove the "Views" folder
- reconnect
- display all objects in the database
When I run this, even though I delete "Object 1", after re-connecting to the database it re-appears. Also as the code also shows it only re-appears in the view as a call to DB.Fetch(docid) is unable to retrieve the object.
I added artificial waits throughout the code to make sure that the data has been written,s o it shouldn't just be a simple race condition.
Does this help?
using System;
using System.Linq;
using System.IO;
using RaptorDB;
namespace testRaptorDB
{
class Program
{
static int waitTime = 5;
static string dataPath = "_data";
static RaptorDB.RaptorDB DB = null;
static Sample[] SampleArray =
{
new Sample
{
ID = Guid.NewGuid(),
Name = "Object 1",
OtherValue = "other value for object 1"
},
new Sample
{
ID = Guid.NewGuid(),
Name = "Object 2",
OtherValue = "other value for object 2"
}
};
public static void Main(string[] args)
{
Console.WriteLine("Delete database directory if it already exists");
Directory.Delete(dataPath, true);
DBInit();
WriteSampleData();
Wait();
ReadAllObjects();
if(DB.Delete(SampleArray[0].ID))
Console.WriteLine("Deleted {0}", SampleArray[0].Name);
else
{
Console.WriteLine("Unable to delete object");
return;
}
Wait();
ReadAllObjects();
RemoveViewFolder();
Wait();
ReadAllObjects();
Console.Write("Press any key to continue . . . ");
Console.ReadKey(true);
}
static void DBInit()
{
Console.WriteLine("Initialise database");
DB = RaptorDB.RaptorDB.Open(dataPath);
DB.RegisterView( new SampleView());
}
static void WriteSampleData()
{
foreach(Sample obj in SampleArray)
{
Console.WriteLine("{0} : {1} : {2}", obj.ID.ToString(), obj.Name, obj.OtherValue);
DB.Save(obj.ID, obj);
}
}
static void ReadAllObjects()
{
Console.WriteLine("Read back all data from DB");
var list = DB.Query("SampleView");
foreach(SampleView.RowSchema row in list.Rows)
{
Console.WriteLine("ViewRow {0}: {1}", row.docid.ToString(), row.Name);
Sample obj = (Sample)DB.Fetch(row.docid);
if(obj != null)
Console.WriteLine("Object {0} : {1} : {2}", obj.ID.ToString(), obj.Name, obj.OtherValue);
else
Console.WriteLine("Can't retrieve original Object");
Console.WriteLine("");
}
}
static void Wait()
{
Console.Write("\nWait a few seconds to make sure records are written");
for(int i =0; i< waitTime; i++)
{
Console.Write(".");
System.Threading.Thread.Sleep(1000);
}
Console.WriteLine("\n");
}
static void RemoveViewFolder()
{
Console.WriteLine("Close connection to DB");
DB.Shutdown();
DB = null;
Console.WriteLine("Remove 'Views' folder from DB folder to force the views to be re-populated");
Directory.Delete(dataPath + @"\Views", true);
Console.WriteLine("Re-connect to database");
DBInit();
}
}
public class Sample
{
public Guid ID;
public string Name;
public string OtherValue;
public Sample()
{
}
}
public class SampleView : View<Sample>
{
public class RowSchema : RDBSchema
{
public string Name;
}
public SampleView()
{
this.Name = "SampleView";
this.Description = "A primary view for sample objects";
this.isPrimaryList = true;
this.isActive = true;
this.BackgroundIndexing = true;
this.Schema = typeof(SampleView.RowSchema);
this.AddFireOnTypes(typeof(Sample));
this.Mapper = (api, docid, doc) =>
{
api.EmitObject(docid, doc);
};
}
}
}
|
|
|
|
|
Try setting the BackgroundIndexing = false
Its the man, not the machine - Chuck Yeager
If at first you don't succeed... get a better publicist
If the final destination is death, then we should enjoy every second of the journey.
|
|
|
|
|
Tried that, the test case still shows the same bug
|
|
|
|
|
I spent a bit more time this morning stepping through the code trying to see what's going wrong exactly but I still haven't tracked down the source of this bug.
In summary
- the delete flag is set when the objects are deleted in both the view and the object store
- when rebuilding the view after deleting the "Views" folder, the delete flag appears to be missing in the view
- the objects in the object store stay deleted, this bug is in the view.
- when rebuilding the view there are objects retrieved which have the delete flag set but their docid = Guid.Empty so we don't see them normally.(only seen these by debugging the code)
The test case I posted yesterday shows the bug fairly clearly, is there anything else you need?
modified 14-Nov-12 13:48pm.
|
|
|
|
|
Thanks a lot, I will work on it.
Its the man, not the machine - Chuck Yeager
If at first you don't succeed... get a better publicist
If the final destination is death, then we should enjoy every second of the journey.
|
|
|
|
|
Thanks, let me know if there's any help I can give.
|
|
|
|
|
As I suspected the issue is that when rebuilding the view the Docid column index is not updated synchroniously hence the rebuild can't find and mark the guid as deleted.
I will update the docid index synchronously all the time in the views.
Its the man, not the machine - Chuck Yeager
If at first you don't succeed... get a better publicist
If the final destination is death, then we should enjoy every second of the journey.
|
|
|
|
|
Thanks for looking into this, let me know when there's a fix available and I'll try it out.
|
|
|
|
|
Ah! the bug was actually in the storagefile not in the indexes.
I have uploaded to codeplex so you can check it out.
Unfortunately the fix is not backward compatible with existing data since the storage file changed.
Its the man, not the machine - Chuck Yeager
If at first you don't succeed... get a better publicist
If the final destination is death, then we should enjoy every second of the journey.
|
|
|
|
|
I've tried out the fix and it appears to be working fine. I'll let you know if anything else pops up but it looks good so far.
Thanks for the help!
|
|
|
|
|
Hi,
Just wondering is there any way the Count and Query methods could be made more like typical Linq methods like
int count = rap.Count<SalesInvoice>();
int count = rap.Count<SalesInvoice>(x=>x.Status == 1);
instead of
int count = rap.Count(typeof(SalesInvoice);
int count = rap.Count(typeof(SalesInvoice), (SalesInvoice si)=>(si.Status == 1);
It would also be great if you could do a similar thing with a query an get a typed result set back of the type you queried.
Also it would be nice to be able to do something similar for the load where the result is typed or null.
I have only just started to play with this DB and apart from finding the query syntax a little strange and getting me head around views where the data is a sub object of the view i am really loving it.
also in case it is helpful i use this wrapper for you count function
public int Count<T>()
{
if (rap != null)
{
return rap.Count(typeof(T));
}
return 0;
}
|
|
|
|
|
Just an update but from what i can tell even though the samples don't do this for querying you can use the generics but it just does not effect case the result and leaves them in an object collection.
With querying the views is it possible to use the schema classes instead of the view class itself as the type. If so it would seem that would then be possible to queries like
Result<SalesInvoice.RowSchema> result = rap.Query<SalesInvoice.RowSchema>(x=>x.Status == 1);
List<SalesInvoice.RowSchema> rows = result.Rows
It is most likely not possible but it just seems like it would be nicer to have a view that really is just the row schema that you do all your queries etc on and then a more private class or method that does the rest of the configuring of the schema as view.
Basically i am just trying to get a basic simple to use interface for my data layer that wraps RaptorDB for my application so that i can use niject or something like it switch to mock DB or another DB. It is just the way i build most of my apps so i am building my tests for playing with RaptorDB the same way as i would hope to use it.
|
|
|
|
|
The query syntax you stated is interesting, but it would only work for the "primary views", unless I can dereference the main view from the schema as you noticed.
Basically RaptorDB only needs the column names for the filters so yes you can use the main class given the properties used are in the view.
I can see the merits of this style, I might see if it is possible, stay tuned.
Its the man, not the machine - Chuck Yeager
If at first you don't succeed... get a better publicist
If the final destination is death, then we should enjoy every second of the journey.
|
|
|
|
|
>>Basically RaptorDB only needs the column names for the filters so yes you can use the main class given the properties used are in the view.
I still need to have a lot closer look at your code but could it be possible to use 2 names for a view instead of one? What i was thinking is one name is the name on the view config class and the other name is schema classes fully qualified name or something like that so that instead of needing to pass the view config class in just the view schema class could be used as that is the only properties/fields you can search on under that view right? Then the results would also be typed in the result and the result object of type Result which you get from the type set on the generic query method.
>>I can see the merits of this style, I might see if it is possible, stay tuned.
If you do have a chance to see if this is possible in some shape or form that would be great. 
|
|
|
|
|
I have managed to come up with a working version! It is a bit of a hack but maybe you could use this as an idea to implement it a better way in the library where i am more limited because i am outside the library.
This is my working query
GenericResult<AnimeSeriesView.RowSchema> result4 = data.Query<AnimeSeriesView.RowSchema>(a => (a.Name == "Test"));
Guid docId = result4.Rows[0].docid;
This is done via a few interfaces and methods
public GenericResult<T> Query<T>(Expression<Predicate<T>> filter) where T : RowSchemaBase, new()
{
if (rap != null)
{
Result res = rap.Query<T>(new T().GetViewType(), filter);
return new GenericResult<T>(res);
}
return null;
}
public class GenericResult<T>
{
public bool OK { get; set; }
public Exception EX { get; set; }
public int TotalCount { get; set; }
public int Count { get; set; }
public List<T> Rows { get; set; }
public GenericResult(Result result)
{
OK = result.OK;
EX = result.EX;
TotalCount = result.TotalCount;
Count = result.Count;
Rows = result.Rows.Cast<T>().ToList();
}
}
public abstract class RowSchemaBase : RDBSchema
{
public abstract Type GetViewType();
}
And below is my Entity and view i use for my tests
public abstract class EntityBase
{
public Guid UniqueID { get; set; }
public EntityBase()
{
UniqueID = Guid.NewGuid();
}
}
public class AnimeSeries : EntityBase
{
public AnimeSeries():base()
{
}
public string Name { get; set; }
}
[RegisterView]
public class AnimeSeriesView : View<AnimeSeries>
{
public class RowSchema : RowSchemaBase {
[FullText]
public string Name;
public override Type GetViewType()
{
return typeof(AnimeSeriesView);
}
}
public AnimeSeriesView()
{
this.Name = "AnimeSeries";
this.Description = "A primary view for AnimeSeries";
this.isPrimaryList = true;
this.isActive = true;
this.BackgroundIndexing = false;
this.Version = 1;
this.Schema = typeof(AnimeSeriesView.RowSchema);
this.AddFireOnTypes(typeof(AnimeSeries));
this.Mapper = (api, docid, doc) =>
{
api.Emit(docid, doc.Name);
};
}
}
|
|
|
|
|
Nice!!
My only concern is if it will work in server mode... (even if it doesn't it's too nicer a feature to ignore since you type less  )
I can probably whittle it down more so you don't need to write the GetViewType() method.
Great job!
Its the man, not the machine - Chuck Yeager
If at first you don't succeed... get a better publicist
If the final destination is death, then we should enjoy every second of the journey.
|
|
|
|
|
I don't think server mode would be an issue if it is done right. I mean you were almost there with your original implementation. Basically it is just removing the types you need to specify witch gives you the nice query syntax and then also casting the response.
The main thing i see having an issue with this method of query is casting the result if you query the entity directly. I have a version of count that looks like it will work on entity and view row schema but i am not sure i can do that for query as the current results for searching on the entity itself does not return the entity but the main view which would break my casting. That said that is an acceptable loss in my case for nice simple syntax. If done at a lower level in your DB lib my base classes and interfaces probably would not be needed either.
Can't wait to see what you manage to do with this. If you need me to test anything let me know
|
|
|
|
|
The following is working in embedded mode (views are the same, nothing changed):
var q = rap.Query<SalesInvoiceView.RowSchema>(x => x.Serial < 100, 0, 10); string s = q.Rows[0].CustomerName;
The constraint is that you have to supply the schema type for the view you want.
I'm working on the server mode conversion...
Its the man, not the machine - Chuck Yeager
If at first you don't succeed... get a better publicist
If the final destination is death, then we should enjoy every second of the journey.
|
|
|
|
|
Nice!
Thanks for making the change! Willit work for other methods like Count as well?
|
|
|
|
|
|
|
 General  News  Suggestion  Question  Bug  Answer  Joke  Rant  Admin
Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.
|
NoSql, JSON based, Document store database with compiled .net map functions and automatic hybrid bitmap indexing and LINQ query filters (now with standalone Server mode, Backup and Active Restore, Transactions, Server side queries, MonoDroid support)
| Type | Article |
| Licence | CPOL |
| First Posted | 29 Apr 2012 |
| Views | 168,618 |
| Downloads | 5,479 |
| Bookmarked | 243 times |
|
|
Submit an article or tip
Post your Blog
