F# vs C#: a performance comparison with BenchmarkDotNet

[EDIT – 02/05/2017] I received quite a lot of comments regarding this post about how it does not fairly reflect the differences between F# and C# and also suggesting possible performance optimizations. I want to highlight from the get-go that this was done on purpose (this is explained in the post below but was maybe not explicit enough). My goal was to compare the traditional OOP approach of C# with the more FP’ish approach of F# on a real use-case. I left out any performance optimization from both versions because the aim was not to optimize code – but rather to show how both versions compare if we write them in a simple and naive way.


Ever since I became interested in functional programming, I’ve read and heard multiple times something along the lines of “F# (or functional programming in general) is slow and that’s why you should stay with OOP languages that guarantee good performance, like Java, C# or C++”.

Because I am curious by nature, I decided to see for myself if indeed I should sacrifice F# on the altar of performance. Specifically, there are two parts of the sentence above that deserve our attention:

“F# is slow”

What does it mean for a programming language to be slow? How can we measure the slowness of a given language? In what context? On what machine, OS, or architecture? Under what conditions? This is definitely not an easy question. Let’s try to put it in a more reasonable way: Is F# so much slower than other OOP languages like C# that we should automatically refrain from using it?

This is easier to answer already. We can definitely measure if a language is slower than another and we can also judge if the difference in performance is something we can live with. This is precisely what we’ll try to do in this post.

“OOP languages guarantee good performance”

Thankfully this is an easy one to debunk. I’ve seen enough systems written in C# that were riddled with performance issues to know this is not the exclusivity of FP languages. I trust in the C# programmers’ ability to create slow and unusable code, too. The same applies to Java and C++, of course.

Actually, there is one thing OOP languages are ultra fast at: introducing bugs in the codebase! I still remember a discussion from a few years ago with some of my former colleagues: “This module sure doesn’t work, but look at how fast it runs!”.

All jokes aside, I hope you get my point. Some languages may be faster than others, but they do not automatically guarantee you good performance. This is something we, programmers, must take care of. With that said, let’s benchmark!

BenchmarkDotNet to the rescue!

We are lucky! The fantastic BenchmarkDotNet tool supports both C# and F# (and also VB if it’s something you’re into). So we will be using it today. But first, we need to know what we will be measuring. I didn’t want to do yet another benchmark comparing the speed of strings concatenation, so I came up with a more realistic business case. We will be comparing the performance between F# and C# for the following tasks:

  • Generate a list of customers
  • Iterate through the list of customers and set as VIP those whose seniority is greater than 2 years.

To have a little more to discuss, we will conduct the experience twice, first with 100 000 customers, and finally with 1 000 000 customers.

Obviously, I wrote the C# and F# samples using different approaches. Therefore:

  • The C# sample uses good ol’ for loops, mutable classes and standard lists.
  • The F# sample uses recursive functions, immutable records and immutable lists.

I wrote both versions in a naive way, with no attempt at performance optimization! Also, before I started benchmarking, I knew that F# would most probably be slower than C# in all cases due to the additional allocations required to achieve immutability, but I had no idea how much slower it would be.

As usual, the full code is available on my GitHub repo here (C# version) and here (F# version).

The C# solution

Let’s have a look at the general skeleton for the C# solution first:

namespace BenchmarkDotNetCsharp
{
    public class Program
    {
        public static void Main(string[] args)
        {
            var summary = BenchmarkRunner.Run<PerformanceTests>();
            Console.WriteLine(summary);
        }
    }

    [MemoryDiagnoser]
    public class PerformanceTests
    {
        private List<Customer> _customers;

        [Benchmark]
        public void CSharp_GenerateCustomers()
        {
            // TODO implement
        }

        [Benchmark]
        public void CSharp_ModifyCustomers()
        {
            // TODO implement
        }
    }

    public class Customer
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public int Age { get; set; }
        public bool IsVip { get; set; }
        public DateTime CustomerSince { get; set; }
    }
}

The structure is really simple. The Program class runs our benchmark using

BenchmarkRunner.Run<PerformanceTests>();

The PerformanceTests class contains the two methods that we want to benchmark, CSharp_GenerateCustomers and CSharp_ModifyCustomers. They are decorated by the Benchmark attribute from BenchmarkDotNet. And finally we have the straightforward Customer class that contains 5 properties.

The customers (respectively 100 000 and 1 000 000) will be held in a standard .NET list:

private List<Customer> _customers;

The F# solution

Now to the F# equivalent:

type Customer = {
    Id: int
    Name: string
    Age: int
    IsVip: bool
    CustomerSince: DateTime
}

[<MemoryDiagnoser>]
type PerformanceTests_ImmutableList() =
    
    let mutable _customers : Customer list = [] // Immutable F# linked-list
   
    [<Benchmark>]
    member this.FSharp_GenerateCustomers_Immutable() =
        // TODO implement

    [<Benchmark>]
    member this.FSharp_ModifyCustomers_Immutable() =
        // TODO implement
  
[<EntryPoint>]
let main argv = 
    BenchmarkRunner.Run<PerformanceTests_ImmutableList>() |> printfn "%A"
    Console.ReadKey() |> ignore
    0

Damn, I love the conciseness of the F# version! Here we have an immutable Customer record type containing the same fields as its C# counterpart, followed by PerformanceTests_ImmutableList. the class we want to benchmark. It contains both FSharp_GenerateCustomers_Immutable and FSharp_ModifyCustomers_Immutable, also decorated by the Benchmark attribute. The customers will be stored in a F# immutable list:

let mutable _customers : Customer list = [] // Immutable F# linked-list

The reference to the list itself is mutable, purely because we want to assign it from the first function and reuse it in the second one.

In case you were wondering, the MemoryDiagnoser attribute from BenchmarkDotNet allows to retrieves additional information about memory (Gen0, Gen1 and Gen2 allocations). This is the setup I used for the benchmarks:

But enough talking, let’s the fight begin!


Generating customers

This is the implementation of the C# method that generates customers:

[Benchmark]
public void CSharp_GenerateCustomers()
{
    var numberToGenerate = 1000000;
    var random = new Random(10);
    var customers = new List<Customer>();
    
    for (int index = 1; index <= numberToGenerate; index++)
    {
        var customer = new Customer()
        {
            Id = index,
            Age = random.Next(18, 100),
            Name = _names[random.Next(0, _names.Length)],
            IsVip = _booleans[random.Next(0, _booleans.Length)],
            CustomerSince = DateTime.Today.AddMonths(-random.Next(0, 121))
        };

        customers.Add(customer);
    }

    _customers = customers;
}

And here’s the F# equivalent:

[<Benchmark>]
member this.FSharp_GenerateCustomers_Immutable() =
    let numberToGenerate = 1000000
    let random = Random(10)
    
    let rec loop customers max current =
        if current > max then
            customers
        else
            let customer = {
                Id = current
                Age = random.Next(18, 100)
                Name = names.[random.Next(0, names.Length)]
                IsVip = booleans.[random.Next(0, booleans.Length)]
                CustomerSince = DateTime.Today.AddMonths(-random.Next(0, 121))
            }

            loop (customer::customers) max (current+1)
       
    // Assign the results to our F# immutable list of customers
    _customers <- loop [] numberToGenerate 1

In both cases, the code produces the exact same data (the Random instance uses the same seed).

Results for 100 000 customers

Below are the results for the generation of 100 000 customers:

And this is the legend for all columns:As you can see the performance is almost the same: 98.4ms for C# versus 101.6ms for F#. We can also notice that there’s a lot more GC collections on F#’s side, but fortunately no Gen2 collection in sight!

I call it a tie! Score: C# 1 / 1 F#

Results for 1 000 000 customers

Below are the results for the generation of 1 000 000 customers:

Here we obtain the following  timings: 1.01s for C# versus 1.12s for F#. F# is almost 11% slower than C# in that specific case. Nothing too bad but a difference still.

Let’s give the point to C# here. Score: C# 2 / 1 F#

Modifying customers

Below is the C# code that iterates through the list and updates customers if they have a seniority greater than 2 years:

[Benchmark]
public void CSharp_ModifyCustomers()
{
    var today = DateTime.Today;

    foreach (var customer in _customers)
    {
        // If customer for more than 2 years
        if ((today - customer.CustomerSince).TotalDays > 365*2)
        {
            customer.IsVip = true;
        }
    }
}

Straightforward. Here’s the F# version:

let setVip today customer =
    match (today - customer.CustomerSince).TotalDays > 365. * 2. with
    | true -> { customer with IsVip = true }
    | false -> customer

[<Benchmark>]
member this.FSharp_ModifyCustomers_Immutable() =
    let today = DateTime.Today

    _customers
    |> List.map (setVip today)
    |> ignore

As F# Customers are immutable, we need to return a new instance every time we need to update IsVip. This is done here:

// ...
| true -> { customer with IsVip = true }
// ...

Results for 100 000 customers

Let’s check the results for 100 000 customers first:

As I suspected, the difference starts to show here. At 4,61ms vs 1,40ms, F# is almost 3.3 times slower than C#. The most significant difference is on the memory side though: zero collections for C# (the modifications are made in-place) while F# has quite a lot of Gen0 and Gen1 collections! This is due to the immutability of both the F# list and the Customer type.

Another point for C#!
Score: C# 3 / 1 F#

Results for 1 000 000 customers

This time we go iterate through the list of 1 000 000 customers!

Now, that’s a huge difference! 14.21ms for C# versus 212.22ms for F#. This time F# is almost 15 times slower than C#, with again a significant amount of Gen0 and Gen1 collections. This is a clear win for C#!

Final Score: C# 4 / 1 F#

Analyzing the results

Now that we have a slightly better view on the topic, let’s go back to our original concern then: Is F# so much slower than other OOP languages like C# that we should automatically refrain from using it?

The first part of the answer is Yes, F# is slower than C# (at least in the scenarios that we saw above).

Now, would the figures above discourage me from using F#? My answer is No. Except for the last test where the difference was pretty significant, I wouldn’t sweat too much over the slight performance hit on the F# side.This is something I am willing to sacrifice in return for the better correctness and readability that F# brings to the table. Not to mention that immutability makes it much easier to introduce concurrency in your application, if you ever need to scale out.

Also, we are only talking about a few milliseconds difference for a simple and naive implementation. In a real business application, this would be completely drown in the total time dedicated to IO operations. You should spend your time on optimizing those instead!

Final words

Out of curiosity, I decided to run once again the last test with 1 000 000 customers in F#, but this time by swapping the immutable list for the .NET standard list (just like in the C# version):

let mutable _customers  = List<Customer>() // Mutable standard .NET list

[<Benchmark>]
member this.FSharp_ModifyCustomers_Mutable() =
    let today = DateTime.Today

    let rec loop index (customers:List<Customer>) =
        if index >= customers.Count
        then ignore
        else 
            customers.[index] <- setVip today customers.[index]
            loop (index+1) customers

    loop 0 _customers |> ignore

And here are the results:As you can see, the results are much better, almost twice as fast than before. This shows that there is place for improvement, but it comes with a trade-off.

That’s it for today. If you have anecdotes or data to share about performance in a functional code base, I’d be more than happy to hear about it!

Until next time!

3 thoughts on “F# vs C#: a performance comparison with BenchmarkDotNet

  1. Pingback: A simple benchmark to compare F# and C# approaches in terms of performance - How to Code .NET

  2. Pingback: dotnetomaniak.pl

Leave a Reply

Your email address will not be published. Required fields are marked *