Maxime FRAPPAT

Hum …no thanks ! – Lordinaire

Author: Maxime FRAPPAT (Page 2 of 19)

Distributed computing with Qarnot

Did you know that your computing tasks can be used as heating ? That’s the goal of the french company named Qarnot who provide computing heater.

A-MA-ZING idea !

Those tasks can be financial computing, Blender rendering, neural network treatment, image processing, …

How it works ?

Basically, you put your ressource (file, scripts, …) in a disk, run a task, get the result in the disk. Simple. The beauty of Qarnot is the ability to work with Docker image (custom or not) with an extreme facility. We will take a look at that later.

https://computing.qarnot.com/developers/overview/qarnot-computing-basics © QarnotTo setup all the process, you can do everything you want with the REST API or the SDK (using Python or Node.js).

Our first task : image processing

To test all that stuff, we will create a little program in .NET Core (<3) using a Docker image and the Python SDK. Our program will apply some filters to an image and returns the result in a new image.

The .NET Core project

Open Visual Studio 2017 and create a new .NET Core Console project called Rendering. Add the Nuget package SixLabors.ImageSharp :

PM> Install-Package SixLabors.ImageSharp -Version 1.0.0-beta0002

Then replace the code in Program.cs by this one :

using SixLabors.ImageSharp;

namespace Rendering
{
    class Program
    {
        static void Main(string[] args)
        {
            using (Image<Rgba32> image = Image.Load(args[0]))
            {
                image.Mutate(x =&amp;gt; x
                    .OilPaint()
                    .Grayscale());
                image.Save($"new_{args[0]}");
            }
        }
    }
}

Publish your app by right-clicking on the project and select Publish…

The Python script :

First step, install Qarnot by following that guide. Note : Clone the repository before starting everything !

When you are ready, create a new file named test.py with that script :

#!/usr/bin/env python
import sys
import qarnot

# Build the full command line
dotnet_cmd = 'dotnet Rendering.dll test.jpg'

# Create a connection, from which all other objects will be derived
conn = qarnot.Connection('qarnot.conf')

# Create a task. The 'with' statement ensures that the task will be
# deleted in the end, to prevent tasks from continuing to run after
# a Ctrl-C for instance
task = conn.create_task('processing-image', 'docker-batch', 1)

# Store if an error happened during the process
error_happened = False

try:
    # Set the command to run when launching the container, by overriding a
    # constant.
    # Task constants are the main way of controlling a task's behaviour
    task.constants['DOCKER_REPO'] = 'microsoft/dotnet'
    task.constants['DOCKER_TAG'] = 'latest'
    task.constants['DOCKER_CMD'] = dotnet_cmd

    # Create a resource disk and add our input file.
    input_disk = conn.create_disk('processing-image-resource')
    input_disk.add_directory("bin")
    input_disk.add_file("test.jpg")

    # Attach the disk to the task
    task.resources.append(input_disk)

    # Submit the task to the Api, that will launch it on the cluster
    task.submit()

    # Wait for the task to be finished, and monitor the progress of its
    # deployment
    last_state = ''
    done = False
    while not done:
        if task.state != last_state:
            last_state = task.state
            print("** {}".format(last_state))

        # Wait for the task to complete, with a timeout of 5 seconds.
        # This will return True as soon as the task is complete, or False
        # after the timeout.
        done = task.wait(60)

        # Display fresh stdout / stderr
        sys.stdout.write(task.fresh_stdout())
        sys.stderr.write(task.fresh_stderr())

    if task.state == 'Failure':
        # Display errors on failure
        print("** Errors: %s" % task.errors[0])
        error_happened = True
    else:
        # Or download the results
        task.download_results('.')

finally:
    task.delete(purge_resources=True, purge_results=True)
    # Exit code in case of error
    if error_happened:
        sys.exit(1)

Well, few things to explain :

# Create a connection, from which all other objects will be derived
conn = qarnot.Connection('qarnot.conf')

You need to create the ‘qarnot.conf’ file to store your Token API.

[cluster]
# url of the REST API
url=https://api.qarnot.com
[client]
# auth string of the client
token=__YOUR_TOKEN_API__

Here is my folders structure, where bin contains the .NET Core console app and qarnot the pyhon SDK. And yes, test.jpg is the image you want to process ;)

It’s time to execute our script and show the result !

$> python test.py
** Submitted
** FullyDispatched
 0> Tue Feb 27 09:02:48 CET 2018
$>

Qarnot provide a monitoring portal that allow you to show all your tasks and disks.

At the end of the execution, you now have a new image named new_test.jpg in your root folder.

 

 

 

 

 

 

 

Sources : https://github.com/Lordinaire/Qarnot-NetCore

Intersection between two lists of dates

I faced a problem when I tried to get the intersection between two lists of DateTime. I wanted to have the intersection only by comparing dates and not dates and time. If I use the LINQ method Intersect on my lists, it will not fit what I want because if the date match but not the time (like different seconds) it will not be part of the intersection.

Of course, I can make my own Intersect method that will test only the date but it’s a little bit overkill. The easy way is to use an IEqualityComparer and pass it to the Intersect method. Here is the implementation I used :

public class DateEqualityComparer : IEqualityComparer<DateTime>
{
    public bool Equals(DateTime date1, DateTime date2)
    {
        return GetHashCode(date1) == GetHashCode(date2);
    }

    public int GetHashCode(DateTime date)
    {
        var hashCode = date.Year ^ date.Month ^ date.Day;
        return hashCode.GetHashCode();
    }
}

A little sample of the case :

public void IntersectDates()
{
    var dates1 = new List<DateTime>
    {
        DateTime.Now,
        DateTime.Now.AddDays(1).AddSeconds(5),
        DateTime.Now.AddDays(3).AddSeconds(10)
    };
    // 05/10/2017 21:40:24
    // 06/10/2017 21:40:29
    // 08/10/2017 21:40:34

    var dates2 = new List<DateTime>
    {
        DateTime.Now,
        DateTime.Now.AddDays(2).AddSeconds(15),
        DateTime.Now.AddDays(3).AddSeconds(20)
    };
    // 05/10/2017 21:40:24
    // 07/10/2017 21:40:39
    // 08/10/2017 21:40:44

    var dateEqualityComparer = new DateEqualityComparer();
    var intersectList = dates1.Intersect(dates2, dateEqualityComparer).ToList();
    Log(intersectList);
}

private static void Log(IEnumerable<DateTime> list)
{
    foreach (var someClass in list)
    {
        Console.WriteLine(someClass.ToString());
    }
    // 05/10/2017
    // 08/10/2017
}

Page 2 of 19

Powered by WordPress & Theme by Anders Norén

%d bloggers like this: