In my case i was working on a query that inner joined three tables together, then grouped by one of the columns in the joined tables. Now LINQ can do this no problem with Join/Join/GroupBy. The difficulty lies in how you project your grouped columns. By default LINQ will just include the whole collection of grouped columns per group row. While this works great in memory, the SQL bridge butchers your call. The reason is that SQL (SQLite at least) can’t do what you asked so the translation is to group by, then do a select all for each group row. As you can imagine, the computational power required grows exponentially with the data size. This is quite clearly very bad.

When working in pure SQL, we can only include columns that are in the group key and columns with an aggregation function applied to them. The aggregation functions are not super obvious on how to use them, so this is what i will be simplifying in this post. Usually what you want is the Sum, Count, First Row, Last Row, etc… from your grouped rows. The main problem is that you can’t just call the LINQ aggregate functions normally, because it doesn’t make sense in the context of your group by projection. I’ll start off with some sample code and move on from there explaining as i go.

First we join the tables together, and project the join into an anonymous type with both values
var q1 = Table1.Join(Table2, x => x.t2key, x => x.key, (t1, t2) => new { t1, t2 })

Now we can group by (say by t1) so that we end up with t1, and an enumerable of t2’s.
var q2 = q1.GroupBy(x => x.t1.key, x => x.t2, (k, x) => new { k.Name, Timestamp = x.Max(y => y.Timestamp), Value = x.Max(y => y.Value) })
so the first lambda tells us what to group on (our parent key), the second lambda describes how to project the grouped data (we will just project the whole t2 table), and the third lambda describes how to project the result set. The third lambda is where things get interesting, we can include members of the key as is (k.Name), but for members of the grouped data, we must apply the aggregate functions or risk a horribly inefficient query. In the above example, i am using the Max aggregate function, which (in SQLite land at least, but others as well im sure) will select the last item in the group as determined by the order by clause (which i didn’t include). Because i didn’t include an order by then its just the last item in the grouped data however the grouped data was built. The order by must appear before the group by in order for it to affect the aggregation functions, so above i would do something like q1 = q1.OrderBy(x => x.t2.Timestamp) in order for the Max function to return the most recent value in the grouped data. If you specify an order by after the group by, it will order the projected group results, so you could do something like q2 = q2.OrderBy(x => x.Value) in order to show the values in ascending order, without affecting the group by aggregation. This is executed by a wrapper query, that uses the projected grouped data as input, so you execute a second query, but that’s it (and this one is generally quite fast)

So the important bits to take a way from this post is that any aggregated columns you want to project must be individually included in your group by result projection, and if you include any enumerations in your projection then they will be converted into subqueries. While my example may not be all that complicated, it is certainly more complicated than simple a select/join/where/order type of LINQ query, which are far more common. By understanding the group by projection core concepts, it should be relatively easy to construct much more complicated (AND efficient) LINQ queries. And remember, when in doubt play around in LINQPad, that program is the best for testing out LINQ queries.

start : [a0, a1, a2], [b0, b1, b2], [c0, c1, c2] –> transpose –> [a0, b0, c0], [a1, b1, c1], [a2, b2, c2]

Now the Zip function in .net4 can do this very easily, but only for TWO enumerables, not any number. Its basic function is to interleave two enumerables and project the result. I wanted to do something similar but without the silly two enumerable limitation. It’s very easy to extend Zip to operate on > 2 enumerables statically, you just need to add more and more params to the extension method, but i wanted to have true dynamic interleaving functionality. So i wrote two extension methods (really just one, and then a simplification for the standard projection) that would do this. One consequence to note is that this will be slower than the built-in Zip, quite obviously because we’re dealing with more enumerables, but also because we can’t run the same type of optimizations that Zip does (this is due to the dynamic nature of this function). So here is some code to look at:

public static IEnumerable<TResult> ZipMany<TItem, TResult>(
    this IEnumerable<IEnumerable<TItem>> source,
    Func<IEnumerable<TItem>, TResult> func)
{
    var iters = source.Select(x => x.GetEnumerator()).ToArray();
    while (iters.All(x => x.MoveNext()))
    {
        yield return func(iters.Select(x => x.Current));
    }
}

public static IEnumerable<List<TItem>> ZipMany<TItem>(
    this IEnumerable<IEnumerable<TItem>> source)
{
    return ZipMany(source, x => x.ToList());
}

So the idea is we build an array of enumerators for each of the input source lists. We have to call ToArray to invoke the execution of the projection, otherwise some strange and undesirable stuff happens. Then we just iterate each enumerator at the same pace and project the index aligned projection of the results. It’s actually pretty simple code for what it is really doing, but it works quite well and i thought i would share it. The second method is just a helper that simplifies the first by removing the need to project the results, this way you can project the results into any custom type if you want, but the default is to simply project them into a list (which is how i can imaging myself and most others wanting the results returned).

I am curious if anyone can point out any improvements or optimizations on this code. Mostly just so i can learn from them myself. Other than that, i am quite happy with how this turned out and it works wonderfully with the application that i wanted to use it with.

This same post is available at the msdn forums if you want to follow the comments there as well: http://social.msdn.microsoft.com/Forums/eu/linqprojectgeneral/thread/40cc2410-adad-4da6-80cd-1736140e885e

The most important thing to note is that you only use the attached property to enable (or disable) a scoped region to share size info within. This is extremely important to understand because scope is not inherited by children if you override it. For example, if you were to create a grid within a grid and put the attached property on both grid nodes, the second grid would override the scope of the first, and you would NOT get your shared sizing. You only ever want to put the attached property ONCE for the ENTIRE scope of shared sizing.

With that out of the way, all you need to do now is give your row or column definitions a unique identifier on the SharedSizeGroup property and then all rows or columns that share the same unique identifier will also share the same hight or width respectively. You should also note that this only works for explicit or Auto sized rows and columns. If you specify a star on a shared item, it will default it to Auto.

When you are creating your DP’s, you have the option of specifying a default (initial) value. The problem with this value is that it is created and stored statically. So if you are creating a DP for, say, a collection, and you initialize it statically in the DP definition then your collection reference now exists statically. So regardless of whether or not you create ten instances of your DO, they will all contain the same initial value because they will all point to the same static reference.

So the conclusion to this quick lesson is that you should never initialize your DP’s with reference types, always perform this step at instantiation to guarantee unique DP values per object instance (unless of course you want to share the reference). As for value types it doesn’t matter nearly as much because the value is copied anyways.

The solution as you may have guessed is to use dynamic types, which allow you to build the type definition at run time and return it.  Since there is no static compilation type checking, it doesn’t care that you can’t define the return type of your method or property.  This isn’t something very common, or even recommended in most situations, but it can come in handy in a pinch.  Here is some totally useless, but good sample code explanation.

private bool flagValue = false;
private bool isFlagValueSet = false;

private dynamic Flag
{
    get
    {
        return new { Value = flagValue, IsValueSet = isFlagValueSet };
    }
}

public bool Test()
{
    return Flag.IsValueSet ? Flag.Value : false;
}

All you gotta do is ensure that you reference the Microsoft.CSharp.dll to get access to dynamic types.  You won’t get any intellisense on the flag variable, but that’s because the type information is computed at runtime only.

The problem here is that we need to bind the DP of the view’s context to multiple sources, something which a standard Binding does not allow.  MultiBinding allows multiple sources, but it would be impossible to know which source was actually “made active” most recently, so we could never return the right source accurately (unless we gave context to the sources themselves, which would be poor design).

The solution to this problem is rather simple, we just need to manage that multi-source aspect of the binding manually in the location where the binding is made or where the DP’s live (this could be the same place if you are doing MVVM).  First create a read only DP that will act as your view driving DP, it can be initialized to null.  Now create a standard event handler function to handle the property changed event for the driving DP on each source.  Hook up the DP value changed events:

private void ViewContext_PropertyChanged(object sender, EventArgs e)
{
    SetValue(ViewContextPropertyKey, (sender as IViewContextProvider).ViewContext);
}

DependencyPropertyDescriptor.FromProperty(Source1ViewModel.ViewContextProperty, typeof(Source1ViewModel))
    .AddValueChanged(m_Source1ViewModel, ViewContext_PropertyChanged);

In the event handler, we simply update your RO DP and that’s it!

If the scenario seems a little obscure, think of this situation.  You want to write a file explorer type of view, the explorer will obviously display files and folders in some sort of list or grid.  On the left hand side, you have multiple panels that determine the context of the file explorer.  One panel is your local system, another panel is a local search, another panel is an FTP site, etc…  All these left side panels can drive your single view through this RO DP pattern.

We have a list of editable items (maybe displayed in a list view), and we have an item editor that can edit a single item.  We want to make our editor handle multiple items in a similar fashion that the visual studio property grid functions.  That is to say that when multiple items are selected, editable fields which are similar across all selected items are displayed, and fields that differ are blank.  The main problem here is that you can’t really databind to a collection of things, not in the sense that we want to solve this problem.  There is no straight forward way of doing this, so my solution is to create an object aggregator class that simulates a single object but in reality handles an enumeration of objects.

I can’t post any code because, while it does work, it is far from polished and i hate to publish unpolished code.  My description of how to do things should be more than enough for anyone to take a solution away from this post.

The Solution

Create a class called ObjectAggregator<T>.  Implement ICustomTypeDescriptor and INotifyPropertyChanged.  Add a public property called Objects which is an Enumerable<T>, this will store our object collection to aggregate.  The constructor should take the object collection and store it in the property.  Now implement the ICustomTypeDescriptor, almost all function and property implementations should look like “return TypeDescriptor.XXX(typeof(T));”  Basically, we are just passing the type description logic onto the actual type we are aggregating.  The exceptions are the GetProperties and GetPropertyOwner functions, this is where we customize things.  GetPropertyOwner just needs to return this, GetProperties is where the real fun occurs.

Get Properties

The empty parameter prototype can just call the other function with null as a parameter (this may be done in the aggregate type’s logic, but do it here just in case).  As for the standard GetProperties(Attribute[]), first get your original list of property descriptors by calling TypeDescriptor.GetProperties(typeof(T), attributes);. Now you can use Cast and Select to generate a list of ObjectAggregatorPropertyDescriptor<T> instances (more on this class below).  Now bundle your list up in a PropertyDescriptorCollection (by using ToArray) and return it.

ObjectAggregatorPropertyDescriptor<T>

This class will inherit from PropertyDescriptor and use the same generic type as the aggregator.  Your constructor will have to take a property descriptor (necessity of the base class), and you should store this paramater locally in a private property.  Now again, most of the property descriptor implementation is pass through.  The only important implementations are where we get or set data, GetValue, SetValue, and ResetValue.  This is because we must aggregate the object collecton held in the type descriptor.  So, most of your implementations should look something like BasePropertyDescriptor.XXX(component).  For the Set and Reset, just cast the component to the aggregator and iterate over the object collection performing the Set or Reset on each object (i.e. BasePropertyDescriptor.Reset(obj)).  As for get, i will post some code because describing LINQ is less fun:

return (component as ObjectAggregator).Objects.Select(x => BasePropertyDescriptor.GetValue(x)).Distinct().SingleOrDefault();

That may look a little crazy, but it’s actually quite simple.  Convert each object into the value returned by the property descriptor, squeeze the list down to unique values and give use the only value (when every object has the same value) or the default for the value type (when values differ between objects).

Thoughts

This is a crappy implementation, but it is dead simple, and it works for most simple use cases.  There are things i didn’t go over which could be considered improvements.  I didn’t talk about INotifyPropertyChanged, but this is pretty straight forward, push out any changes to any objects in the aggregator as if they were its own.  Ideally, all your property values would be Nullable so that value collisions are actually properly handled as a collision (rather than just using the default value), this is especially true for value types where a collision may hinder the experience (think bools).  I can’t remember, but i think calling GetPropertyOwner is ideal as well, it’s been a while since i really dived into custom type descriptors.  Finally, if you wanted to really make this robust, you would handle the scenario of multiple object types.  It’s not too much more code, but probably quite messy if you have name/type collisions, since you would really have to pick how to handle that yourself, there is no default way.  I think the property grid would likely filter out any name/type collisions (i.e. the property “Flag” is a bool on one object and an Enum on another object in the same collection).

Anyways, this is *A* way to solve this problem, not the most elegant, but definitely on the path to the most elegant.  This would definitely need some work to function in a more complicated scenario (with converters, editors, attributes, etc…).  But in the end, it does (poorly) what i intended it to do, that is allow me to databind a single field to an object collection.  I would love to hear of any other ways to accomplish this task.

This will add my rtorrent ppa to your custom sources, update your source cache, then install the updated version of rtorrent (as well as libtorrent and xmlrpc). Combine this with rutorrent and you have a pretty sweet headless torrent server.

first, some things you will need to install:
sudo apt-get install dput devscripts
of course, you’ll need some of the usual stuff like build-essentials and other common dev packages. dput is used to upload your source packages to your ppa, and devscripts contains some useful debian package scripts, such as debuild which is used to build your package.

now, create a gpg key for your ppa (or use an existing one if you have one)
simply use gpg –gen-key to generate a key (use the defaults, unless you want to get more specific)
if you already have a set of keys on another machine, you can export/import the public and private keys to your other machine. If you get into trouble with not enough entropy (happens when your compile machine is headless and you do everything through ssh), then try installing rng-tools and set your device to /dev/urandom (in /etc/default/rng-tools) then start the service. This will generate entropy for you.

Now setup your own ppa. First you’ll need a launchpad account, then you just need to authenticate yourself and create a new ppa. I won’t detail this process, but you can probably get a good idea from here: http://blog.launchpad.net/ppa/personal-package-archives-for-everyone

Now grab your source package as well as its build dependencies:
sudo apt-get build-dep PACKAGE
apt-get source PACKAGE
note that you don’t need to sudo when getting package sources. If you are upgrading the package using the latest version, things can get a little wonky because ubuntu packages are usually customized with patches, and these patches often don’t work with bleeding edge source. None the less you can give it a shot, simply grab the source for your package (not through apt, but from svn, git, http, etc…) and then copy the debian directory from the ubuntu source directory to your newer source directory. You will need to modify the changelog and create a new entry for your package (with an updated version). Alternatively, you can use dch to do this. Once you have your source set up, you have to build the source package. I recommend building the binary first, but this is not necessary, it just helps to for testing installation of the package and other things. to build the source package:
debuild -S -sa
this will generate a source package in the parent directory. Once this finishes, you can upload it to you ppa.

to upload your package to your ppa, just run:
dput PPA PACKAGE_source.changes
PPA will be something like ppa:USER_NAME/PPA_NAME
the changes file will have been created as a part of the source package creation.
this will ask you to enter your gpg password, and then upload the files to the launchpad servers. In a few minutes your package will be built and you’ll get a notice saying it was successful or not.

Caveats! They exist! some things i have found which are slightly annoying are the following:
you cannot re-upload (re-build) a package. This is useful if you have a dependency issue, the only fix is to increment the version and re-upload (please correct me if there is a better solution).
deleting packages is not very clear what happens, a deleted package still exists for quite some time on the server. There appears to be no notice of a package being purged, it just randomly stops existing.

p.s. updated rtorrent ppa will come soon, if you really want to get it now, you can use this package source:
deb http://ppa.launchpad.net/patricksissons/pjs/ubuntu karmic main
this is my development ppa, so it may contain things you don’t actually want to install.