Hum …no thanks ! – Lordinaire

Page 2 of 19

Distributed computing with Qarnot

Did you know that your computing tasks can be used as heating ? That’s the goal of the french company named Qarnot who provide computing heater.

A-MA-ZING idea !

Those tasks can be financial computing, Blender rendering, neural network treatment, image processing, …

How it works ?

Basically, you put your ressource (file, scripts, …) in a disk, run a task, get the result in the disk. Simple. The beauty of Qarnot is the ability to work with Docker image (custom or not) with an extreme facility. We will take a look at that later. © QarnotTo setup all the process, you can do everything you want with the REST API or the SDK (using Python or Node.js).

Our first task : image processing

To test all that stuff, we will create a little program in .NET Core (<3) using a Docker image and the Python SDK. Our program will apply some filters to an image and returns the result in a new image.

The .NET Core project

Open Visual Studio 2017 and create a new .NET Core Console project called Rendering. Add the Nuget package SixLabors.ImageSharp :

PM> Install-Package SixLabors.ImageSharp -Version 1.0.0-beta0002

Then replace the code in Program.cs by this one :

using SixLabors.ImageSharp;

namespace Rendering
    class Program
        static void Main(string[] args)
            using (Image<Rgba32> image = Image.Load(args[0]))
                image.Mutate(x =&amp;gt; x

Publish your app by right-clicking on the project and select Publish…

The Python script :

First step, install Qarnot by following that guide. Note : Clone the repository before starting everything !

When you are ready, create a new file named with that script :

#!/usr/bin/env python
import sys
import qarnot

# Build the full command line
dotnet_cmd = 'dotnet Rendering.dll test.jpg'

# Create a connection, from which all other objects will be derived
conn = qarnot.Connection('qarnot.conf')

# Create a task. The 'with' statement ensures that the task will be
# deleted in the end, to prevent tasks from continuing to run after
# a Ctrl-C for instance
task = conn.create_task('processing-image', 'docker-batch', 1)

# Store if an error happened during the process
error_happened = False

    # Set the command to run when launching the container, by overriding a
    # constant.
    # Task constants are the main way of controlling a task's behaviour
    task.constants['DOCKER_REPO'] = 'microsoft/dotnet'
    task.constants['DOCKER_TAG'] = 'latest'
    task.constants['DOCKER_CMD'] = dotnet_cmd

    # Create a resource disk and add our input file.
    input_disk = conn.create_disk('processing-image-resource')

    # Attach the disk to the task

    # Submit the task to the Api, that will launch it on the cluster

    # Wait for the task to be finished, and monitor the progress of its
    # deployment
    last_state = ''
    done = False
    while not done:
        if task.state != last_state:
            last_state = task.state
            print("** {}".format(last_state))

        # Wait for the task to complete, with a timeout of 5 seconds.
        # This will return True as soon as the task is complete, or False
        # after the timeout.
        done = task.wait(60)

        # Display fresh stdout / stderr

    if task.state == 'Failure':
        # Display errors on failure
        print("** Errors: %s" % task.errors[0])
        error_happened = True
        # Or download the results

    task.delete(purge_resources=True, purge_results=True)
    # Exit code in case of error
    if error_happened:

Well, few things to explain :

# Create a connection, from which all other objects will be derived
conn = qarnot.Connection('qarnot.conf')

You need to create the ‘qarnot.conf’ file to store your Token API.

# url of the REST API
# auth string of the client

Here is my folders structure, where bin contains the .NET Core console app and qarnot the pyhon SDK. And yes, test.jpg is the image you want to process ;)

It’s time to execute our script and show the result !

$> python
** Submitted
** FullyDispatched
 0> Tue Feb 27 09:02:48 CET 2018

Qarnot provide a monitoring portal that allow you to show all your tasks and disks.

At the end of the execution, you now have a new image named new_test.jpg in your root folder.








Sources :

Intersection between two lists of dates

I faced a problem when I tried to get the intersection between two lists of DateTime. I wanted to have the intersection only by comparing dates and not dates and time. If I use the LINQ method Intersect on my lists, it will not fit what I want because if the date match but not the time (like different seconds) it will not be part of the intersection.

Of course, I can make my own Intersect method that will test only the date but it’s a little bit overkill. The easy way is to use an IEqualityComparer and pass it to the Intersect method. Here is the implementation I used :

public class DateEqualityComparer : IEqualityComparer<DateTime>
    public bool Equals(DateTime date1, DateTime date2)
        return GetHashCode(date1) == GetHashCode(date2);

    public int GetHashCode(DateTime date)
        var hashCode = date.Year ^ date.Month ^ date.Day;
        return hashCode.GetHashCode();

A little sample of the case :

public void IntersectDates()
    var dates1 = new List<DateTime>
    // 05/10/2017 21:40:24
    // 06/10/2017 21:40:29
    // 08/10/2017 21:40:34

    var dates2 = new List<DateTime>
    // 05/10/2017 21:40:24
    // 07/10/2017 21:40:39
    // 08/10/2017 21:40:44

    var dateEqualityComparer = new DateEqualityComparer();
    var intersectList = dates1.Intersect(dates2, dateEqualityComparer).ToList();

private static void Log(IEnumerable<DateTime> list)
    foreach (var someClass in list)
    // 05/10/2017
    // 08/10/2017

Page 2 of 19

Powered by WordPress & Theme by Anders Norén

%d bloggers like this: