BASTA Spring 2013 - Custom OData Providers

27 February 2013 - .NET, Azure, C#

Introduction

At the BASTA Spring 2013 conference I had a session about developing custom OData providers. In this blog article I summarize the most important take aways and publish the source code. For completeness here is the abstract of the talk in German:

In seiner OData Session zeigt Rainer Stropek, wie man eigene OData-Provider entwickelt. In einem durchgängigen Beispiel demonstriert er, wie man erst einen LINQ-Provider und darauf aufbauend einen OData-konformen REST Service erstellt und von verschiedenen Programmiersprachen und Tools darauf zugreift. In der Session werden Grundkenntnisse von OData und LINQ vorausgesetzt.

Note that I have done multiple talks about custom OData providers in the past. Amongst others I did talks about a concrete application of a custom OData provider: Fan-out queries in Sharding-scenarios. You can read more about it in this blog article (2011).

Slides

BASTA 2013: Custom OData Provider from Rainer Stropek

Download Sourcecode

Introducing OData

OData is a great protocol for web-enabling CRUD scenarios. It is a platform-independent standard. You can get client and server libraries for a bunch of different platforms. For more information about the protocol see OData homepage. .NET contains support for OData for quite a long time. Recently Microsoft has begun to ship its latest OData implementation using Nuget packges. For our sample you need to get WCF Data Services Server and its dependencies.

The Basics

Most of Microsoft's cloud services (e.g. Table Storage) support OData out of the box. You do not need to implement an OData server yourself. Unfortunately SQL Server does not support OData natively - neither on-premise nor in the cloud. However, that's not a big problem in practise. If you have an Entity Framework model, Microsoft's OData SDK can turn it into an OData feed with just a few lines of code. If you want to learn more about it, take a look at this how-to chapter in MSDN. We will not go into details on this in this session.

In our sample we want to start with a simple OData service backed by an in-memory collection of Customer objects. The following code snippets show the Customer class and a helper class used to generate demo data. This code is just infrastructure, it is not specific to OData.

using System;
using System.Collections.Generic;
using System.Linq;

namespace CustomLinqProvider
{
    public class Customer
    {
        public int CustomerID { get; set; }
        public string CompanyName { get; set; }
        public string ContactPersonFirstName { get; set; }
        public string ContactPersonLastName { get; set; }

        public override string ToString()
        {
            return string.Format("{0}, {1} {2}, {3}", this.CustomerID, this.ContactPersonFirstName, this.ContactPersonLastName, this.CompanyName);
        }

        public static IReadOnlyList<Customer> GenerateDemoCustomers(int firstCustomerID = 0, int numberOfCustomers = 100)
        {
            var rand = new Random();
            return Enumerable.Range(firstCustomerID, numberOfCustomers)
                .Select(i => new Customer()
                {
                    CustomerID = i,
                    ContactPersonLastName = DemoNames.LastNames[rand.Next(DemoNames.LastNames.Count)],
                    ContactPersonFirstName = DemoNames.FirstNames[rand.Next(DemoNames.FirstNames.Count)],
                    CompanyName = string.Format(
                        "{0} {1} {2}",
                        DemoNames.CompanyNamesPart1[rand.Next(DemoNames.CompanyNamesPart1.Count)],
                        DemoNames.CompanyNamesPart2[rand.Next(DemoNames.CompanyNamesPart2.Count)],
                        DemoNames.CompanyNamesPart3[rand.Next(DemoNames.CompanyNamesPart3.Count)])
                })
                .ToArray();
        }
    }
}

using System.Collections.Generic;

namespace CustomLinqProvider
{
    /// <summary>
    /// Contains some common names used to generate customer demo data
    /// </summary>
    public static class DemoNames
    {
        public static readonly IReadOnlyList<string> LastNames = new [] {
            "Smith", "Johnson", "Williams", "Jones", "Brown", "Davis", "Miller", "Wilson", "Moore", "Taylor" /*, ...*/
        };

        public static readonly IReadOnlyList<string> FirstNames = new[] {
            "Jack", "Lewis", "Riley", "James", "Logan" /*, ...*/
        };

        public static readonly IReadOnlyList<string> CompanyNamesPart1 = new[] { 
            "Corina", "Amelia", "Menno", "Malthe", "Hartwing", "Marlen" /*, ...*/ };
        public static readonly IReadOnlyList<string> CompanyNamesPart2 = new[] { 
            "Construction", "Engineering", "Consulting", "Trading", "Metal Construction", "Publishers"  /*, ...*/ };
        public static readonly IReadOnlyList<string> CompanyNamesPart3 = new[] {
            "Ltd", "Limited", "Corporation", "Limited Company", "Joint Venture", "Ltd.", "Cooperative" };
    }
}

Publishing generated customer demo data with OData is really simple. Here is the code you need for it:

<%@ ServiceHost Language="C#" Factory="System.Data.Services.DataServiceHostFactory, Microsoft.Data.Services, Version=5.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" Service="CustomODataService.CustomerReflectionService" %>

using IQToolkit;
using CustomLinqProvider;
using System.Data.Services;
using System.Data.Services.Common;
using System.Linq;
using System.ServiceModel;

namespace CustomODataService
{
    /// <summary>
    /// Implements a context class that contain queryables which we want to expose using OData
    /// </summary>
    public class CustomerReflectionContext 
    {
        public IQueryable<Customer> Customer
        {
            get
            {
                // Generate 1000 customers and return queryable so that user can query the
                // generated data.
                return CustomLinqProvider.Customer
                    .GenerateDemoCustomers(numberOfCustomers: 100)
                    .AsQueryable();
            }
        }
    }
    [ServiceBehavior(IncludeExceptionDetailInFaults = true)]
    public class CustomerReflectionService : DataService<CustomerReflectionContext>
    {
        // This method is called only once to initialize service-wide policies.
        public static void InitializeService(DataServiceConfiguration config)
        {
            // TODO: set rules to indicate which entity sets and service operations are visible, updatable, etc.
            // Examples:
            config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);
            // config.SetServiceOperationAccessRule("MyServiceOperation", ServiceOperationRights.All);
            config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
            config.UseVerboseErrors = true;
        }
    }
}

You can immediately try your OData service in the browser (click to enlarge the image):

Your OData service will also support JSON. You can try that by specifying an appropriate accept-header in Fiddler (click to enlarge the image):

Custom LINQ Provider

Did you recognize the problem of the code shown above? The class CustomerReflectionContext always generates 100 customer objects. An OData client can specify a $top clause as shown above. However, the server will still generate 100 objects. Somewhere in the OData stack the unwanted 99 customers will be thrown away. Imagine generating the customer objects would be a difficult and time-consuming process. In such a case an architecture that works like the classes shown above would not work.

A solution to this problem is a custom LINQ provider. A LINQ provider takes an expression tree and has to interpret it. In our case we will just recognize the Take and Skip methods of the expression tree. We will use them to limit the number of customers to generate. Implementing a custom LINQ provider from scratch is hard. Fortunately there are multiple helper libraries available. In my sample I use the IQToolkit library. With its help, implementing the custom LINQ provider is easy. Here is the code:

using System;
using System.Linq.Expressions;
using System.Reflection;

namespace CustomLinqProvider
{
    /// <summary>
    /// Simple visitor that extracts "Take" and "Skip" clauses from expression tree
    /// </summary>
    internal class AnalyzeQueryVisitor : ExpressionVisitor
    {
        public AnalyzeQueryVisitor()
        {
            this.Take = 100;
            this.Skip = 0;
        }

        public int Take { get; private set; }
        public int Skip { get; private set; }

        protected override Expression VisitMethodCall(MethodCallExpression m)
        {
            switch (m.Method.Name)
            {
                case "Take":
                    this.Take = (int)(m.Arguments[1] as ConstantExpression).Value;
                    break;
                case "Skip":
                    this.Skip = (int)(m.Arguments[1] as ConstantExpression).Value;
                    break;
                case "OrderBy":
                    // We do not check/consider order by yet.
                    break;
                default:
                    throw new CustomLinqProviderException("Method not supported!");
            }

            return base.VisitMethodCall(m);
        }
    }
}

using IQToolkit;
using System;
using System.Linq;
using System.Linq.Expressions;

namespace CustomLinqProvider
{
    public class DemoCustomerProvider : QueryProvider
    {
        public override object Execute(Expression expression)
        {
            // Use a visitor to extract demo data generation parameters
            // ("take" and "skip" clauses)
            var analyzer = new AnalyzeQueryVisitor();
            analyzer.Visit(expression);

            // Generate data
            return Customer.GenerateDemoCustomers(analyzer.Skip, analyzer.Take);
        }

        public override string GetQueryText(Expression expression)
        {
            throw new NotImplementedException();
        }
    }
}

using System;
using System.Runtime.Serialization;

namespace CustomLinqProvider
{
    [Serializable]
    public class CustomLinqProviderException : NotSupportedException
    {
        public CustomLinqProviderException()
            : base()
        {
        }

        public CustomLinqProviderException(string message)
            : base(message)
        {
        }

        protected CustomLinqProviderException(SerializationInfo info, StreamingContext context)
            : base(info, context)
        {
        }

        public CustomLinqProviderException(string message, Exception innerException)
            : base(message, innerException)
        {
        }
    }
}

Of course you do not absolutely need OData to make use of the LINQ provider. You can use it directly in your C# app, too. You can easily try it e.g. in a unit test:

using IQToolkit;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using CustomLinqProvider;
using System.Linq;

namespace CustomODataProvider.Test
{
    [TestClass]
    public class LinqProviderTest
    {
        [TestMethod]
        public void TestSuccessfullQueries()
        {
            var provider = new DemoCustomerProvider();

            var result = new Query<Customer>(provider).ToArray();
            Assert.AreEqual(100, result.Length);
            Assert.AreEqual(0, result[0].CustomerID);

            result = new Query<Customer>(provider).Skip(100).ToArray();
            Assert.AreEqual(100, result.Length);
            Assert.AreEqual(100, result[0].CustomerID);

            result = new Query<Customer>(provider).Skip(100).Take(10).ToArray();
            Assert.AreEqual(10, result.Length);
            Assert.AreEqual(100, result[0].CustomerID);
        }

        [TestMethod]
        public void TestIllegalQuery()
        {
            var provider = new DemoCustomerProvider();
            bool exception = false;
            try
            {
                new Query<Customer>(provider).Where(c => c.CustomerID == 5).ToArray();
            }
            catch (CustomLinqProviderException)
            {
                exception = true;
            }

            Assert.IsTrue(exception);
        }
    }
}

Using the LINQ Provider in OData

Now that we have the optimized LINQ provider, we can back our OData service with it:

/// <summary>
/// Implements a context class that contain queryables which we want to expose using OData
/// </summary>
public class CustomerReflectionContext 
{
    //public IQueryable<Customer> Customer
    //{
    //  get
    //  {
    //      // Generate 1000 customers and return queryable so that user can query the
    //      // generated data.
    //      return CustomLinqProvider.Customer
    //          .GenerateDemoCustomers(numberOfCustomers: 100)
    //          .AsQueryable();
    //  }
    //}

    public IQueryable<Customer> Customer
    {
        get
        {
            // Use custom linq provider to generate exactly the number of customers we need
            return new Query<Customer>(new DemoCustomerProvider());
        }
    }
}

It you query the OData service with a $top clause now, the service will only generated the request number of customer objects in the background.

Unfortunately we have lost functionality with our customer LINQ provider, too. If you decide to go for a customer provider, you have to deal with all possible LINQ functions yourself. In our case we only allow $top and $skip. In all other cases we throw an exception (click to enlarge image):

Building a Completely Customized OData Provider

The example shown above can already dynamically handle parameters specified in OData queries during runtime. However, there are cases in which you might need even more flexibility. Imagine that you do not know the available entities or properties during compile time. Our own SaaS product time cockpit allows power users to customize its data model. Users can add custom entities, add properties, define calculated properties with an easy-to-learn formula language. In such a case, writing a custom LINQ provider and backing an OData service is not sufficient. We have to dive deeper into Microsoft's OData SDK.

In the sample code shown above we implemented the class CustomerReflectionContext which contains a property of type IQueryable. Our OData service CustomerReflectionService derives from DataService<CustomerReflectionContext>. By specifying the type parameter, Microsoft's OData SDK will look for all IQueryable properties in the specified class. In the custom OData provider sample we want to implement now, we do not know during compile time which IQueryable properties we will have during runtime. Therefore we change from a class that contains a fixed number of properties to a class that can dynamically provide IQueryables during runtime:

using System;
using System.Data.Services;
using System.Data.Services.Common;
using System.Data.Services.Providers;
using CustomODataService.CustomDataServiceBase;
using System.Threading;
using CustomLinqProvider;
using System.Reflection;
using System.Linq;
using IQToolkit;

namespace CustomODataService
{
    public class CustomerServiceDataContext : IGenericDataServiceContext
    {
        public IQueryable GetQueryable(ResourceSet set)
        {
            if (set.Name == "Customer")
            {
                return new Query<Customer>(new DemoCustomerProvider());
            }

            return null;
        }
    }
}

using System.Data.Services.Providers;
using System.Linq;

namespace CustomODataService.CustomDataServiceBase
{
    /// <summary>
    /// Acts as the interface for data service contexts used for a custom data service
    /// </summary>
    public interface IGenericDataServiceContext
    {
        /// <summary>
        /// Creates a queryable for the specified resource set
        /// </summary>
        /// <param name="set">Resource set for which the queryable should be created</param>
        IQueryable GetQueryable(ResourceSet set);
    }
}

Next we have to somehow inform the OData runtime about which types and properties our class can provide. This is done using OData's ResourceSet and ResourceType objects in combination with the interface IDataServiceMetadataProvider. Note that I have kept the following implementation consciously simple. It should demonstrate the concept without you having to worry about use-case-specific implementation details.

using System;
using System.Collections.Generic;
using System.Data.Services.Providers;
using System.Reflection;
using System.Linq;

namespace CustomODataService.CustomDataServiceBase
{
    public class CustomDataServiceMetadataProvider : IDataServiceMetadataProvider
    {
        private Dictionary<string, ResourceType> resourceTypes = new Dictionary<string, ResourceType>();
        private Dictionary<string, ResourceSet> resourceSets = new Dictionary<string, ResourceSet>();

        /// <summary>
        /// Add a resource type
        /// </summary>
        /// <param name="type">Type to add</param>
        public void AddResourceType(ResourceType type)
        {
            type.SetReadOnly();
            resourceTypes.Add(type.FullName, type);
        }

        /// <summary>
        /// Adds the resource set.
        /// </summary>
        /// <param name="set">The set.</param>
        public void AddResourceSet(ResourceSet set)
        {
            set.SetReadOnly();
            resourceSets.Add(set.Name, set);
        }

        public static IDataServiceMetadataProvider BuildDefaultMetadataForClass<TEntity>(string namespaceName)
        {
            // Add resource type for class TEntity
            var productType = new ResourceType(
                typeof(TEntity),
                ResourceTypeKind.EntityType,
                null, // BaseType 
                namespaceName, // Namespace 
                typeof(TEntity).Name,
                false // Abstract? 
            );

            // use reflection to get all primitive properties
            foreach (var property in typeof(TEntity)
                    .GetProperties(BindingFlags.Public | BindingFlags.Instance)
                    .Where(pi => pi.DeclaringType == typeof(TEntity) && (pi.PropertyType.IsPrimitive || pi.PropertyType == typeof(string))))
            {
                var resourceProperty = new ResourceProperty(
                    property.Name,
                    // For simplicity let's assume that the property with postfix ID is the key property
                    ResourcePropertyKind.Primitive | (property.Name.EndsWith("ID") ? ResourcePropertyKind.Key : 0),
                    ResourceType.GetPrimitiveResourceType(property.PropertyType));
                productType.AddProperty(resourceProperty);
            }

            // Build metadata object
            var metadata = new CustomDataServiceMetadataProvider();
            metadata.AddResourceType(productType);
            metadata.AddResourceSet(new ResourceSet(typeof(TEntity).Name, productType));
            return metadata;
        }

        #region Implementation of IDataServiceMetadataProvider
        public string ContainerName
        {
            get { return "Container"; }
        }

        public string ContainerNamespace
        {
            get { return "Namespace"; }
        }

        public IEnumerable<ResourceType> GetDerivedTypes(ResourceType resourceType)
        {
            // We don't support type inheritance yet 
            yield break;
        }

        public ResourceAssociationSet GetResourceAssociationSet(ResourceSet resourceSet, ResourceType resourceType, ResourceProperty resourceProperty)
        {
            throw new NotImplementedException("No relationships.");
        }

        public bool HasDerivedTypes(ResourceType resourceType)
        {
            // We don’t support inheritance yet 
            return false;
        }

        public IEnumerable<ResourceSet> ResourceSets
        {
            get { return this.resourceSets.Values; }
        }

        public IEnumerable<ServiceOperation> ServiceOperations
        {
            // No service operations yet 
            get { yield break; }
        }

        public bool TryResolveResourceSet(string name, out ResourceSet resourceSet)
        {
            return resourceSets.TryGetValue(name, out resourceSet);
        }

        public bool TryResolveResourceType(string name, out ResourceType resourceType)
        {
            return resourceTypes.TryGetValue(name, out resourceType);
        }

        public bool TryResolveServiceOperation(string name, out ServiceOperation serviceOperation)
        {
            // No service operations are supported yet 
            serviceOperation = null;
            return false;
        }

        public IEnumerable<ResourceType> Types
        {
            get { return this.resourceTypes.Values; }
        }
        #endregion
    }
}

We are still missing a class that connects our CustomerServiceDataContext class with OData. This is done using OData's IDataServiceQueryProvider interface. It has to be able to generate an IQueryable for every resource set that we have added to our metadata (see above). Our sample implementation of IDataServiceQueryProvider is again kept simple.

using System;
using System.Collections.Generic;
using System.Data.Services.Providers;
using System.Linq;

namespace CustomODataService.CustomDataServiceBase
{
    public class CustomDataServiceProvider : IDataServiceQueryProvider
    {
        private IGenericDataServiceContext currentDataSource;
        private IDataServiceMetadataProvider metadata;

        public CustomDataServiceProvider(IDataServiceMetadataProvider metadata, IGenericDataServiceContext dataSource)
        {
            this.metadata = metadata;
            this.currentDataSource = dataSource;
        }

        public object CurrentDataSource
        {
            get { return currentDataSource; }
            set { currentDataSource = value as IGenericDataServiceContext; }
        }

        public IQueryable GetQueryRootForResourceSet(ResourceSet resourceSet)
        {
            return currentDataSource.GetQueryable(resourceSet);
        }

        public ResourceType GetResourceType(object target)
        {
            var type = target.GetType();
            return metadata.Types.Single(t => t.InstanceType == type);
        }

        #region Implementation of IDataServiceQueryProvider
        public bool IsNullPropagationRequired
        {
            get { return true; }
        }

        public object GetOpenPropertyValue(object target, string propertyName)
        {
            throw new NotImplementedException();
        }

        public IEnumerable<KeyValuePair<string, object>> GetOpenPropertyValues(object target)
        {
            throw new NotImplementedException();
        }

        public object GetPropertyValue(object target, ResourceProperty resourceProperty)
        {
            throw new NotImplementedException();
        }

        public object InvokeServiceOperation(ServiceOperation serviceOperation, object[] parameters)
        {
            throw new NotImplementedException();
        }
        #endregion
    }
}

That's it. We have all prerequisites to setup our fully customizable OData service. Note how the .NET interface IServiceProvider is used to link our service with the previously created implementations of IDataServiceMetadataProvider and IDataServiceQueryProvider.

<%@ ServiceHost Language="C#" Factory="System.Data.Services.DataServiceHostFactory, Microsoft.Data.Services, Version=5.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" Service="CustomODataService.CustomerService" %>

using System;
using System.Data.Services;
using System.Data.Services.Common;
using System.Data.Services.Providers;
using CustomODataService.CustomDataServiceBase;
using System.Threading;
using CustomLinqProvider;
using System.Reflection;
using System.Linq;
using IQToolkit;

namespace CustomODataService
{
    public class CustomerService : DataService<object>, IServiceProvider
    {
        private readonly IDataServiceMetadataProvider customerMetadata;
        private readonly CustomDataServiceProvider dataSource;

        public CustomerService()
        {
            this.customerMetadata = CustomDataServiceMetadataProvider.BuildDefaultMetadataForClass<Customer>("DefaultNamespace");
            this.dataSource = new CustomDataServiceProvider(this.customerMetadata, new CustomerServiceDataContext());
        }

        public static void InitializeService(DataServiceConfiguration config)
        {
            // Enable read for all entities
            config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);

            // Various other settings
            config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
            config.DataServiceBehavior.AcceptProjectionRequests = false;
        }

        public object GetService(Type serviceType)
        {
            if (serviceType == typeof(IDataServiceMetadataProvider))
            {
                return this.customerMetadata;
            }
            else if (serviceType == typeof(IDataServiceQueryProvider))
            {
                return this.dataSource;
            }

            return null;
        }
    }
}

BASTA Spring 2013 - Custom OData Providers

Introduction

Slides

Download Sourcecode

Introducing OData

The Basics

Custom LINQ Provider

Using the LINQ Provider in OData

Building a Completely Customized OData Provider

Further Readings

Content

Tags

Authors

BASTA Spring 2013 - Custom OData Providers

Introduction

Slides

Download Sourcecode

Introducing OData

The Basics

Custom LINQ Provider

Using the LINQ Provider in OData

Building a Completely Customized OData Provider

Further Readings

Content

Tags

Authors

Related Posts

ICCC - Azure Container Registry

Azure DevCamp Vienna - Docker on Azure

Managed Identities and Azure Key Vault