Maxime FRAPPAT

Hum …no thanks ! – Lordinaire

Tag: Docker (Page 1 of 2)

Distributed computing with Qarnot

Did you know that your computing tasks can be used as heating ? That’s the goal of the french company named Qarnot who provide computing heater.

A-MA-ZING idea !

Those tasks can be financial computing, Blender rendering, neural network treatment, image processing, …

How it works ?

Basically, you put your ressource (file, scripts, …) in a disk, run a task, get the result in the disk. Simple. The beauty of Qarnot is the ability to work with Docker image (custom or not) with an extreme facility. We will take a look at that later.

https://computing.qarnot.com/developers/overview/qarnot-computing-basics © QarnotTo setup all the process, you can do everything you want with the REST API or the SDK (using Python or Node.js).

Our first task : image processing

To test all that stuff, we will create a little program in .NET Core (<3) using a Docker image and the Python SDK. Our program will apply some filters to an image and returns the result in a new image.

The .NET Core project

Open Visual Studio 2017 and create a new .NET Core Console project called Rendering. Add the Nuget package SixLabors.ImageSharp :

PM> Install-Package SixLabors.ImageSharp -Version 1.0.0-beta0002

Then replace the code in Program.cs by this one :

using SixLabors.ImageSharp;

namespace Rendering
{
    class Program
    {
        static void Main(string[] args)
        {
            using (Image<Rgba32> image = Image.Load(args[0]))
            {
                image.Mutate(x =&amp;gt; x
                    .OilPaint()
                    .Grayscale());
                image.Save($"new_{args[0]}");
            }
        }
    }
}

Publish your app by right-clicking on the project and select Publish…

The Python script :

First step, install Qarnot by following that guide. Note : Clone the repository before starting everything !

When you are ready, create a new file named test.py with that script :

#!/usr/bin/env python
import sys
import qarnot

# Build the full command line
dotnet_cmd = 'dotnet Rendering.dll test.jpg'

# Create a connection, from which all other objects will be derived
conn = qarnot.Connection('qarnot.conf')

# Create a task. The 'with' statement ensures that the task will be
# deleted in the end, to prevent tasks from continuing to run after
# a Ctrl-C for instance
task = conn.create_task('processing-image', 'docker-batch', 1)

# Store if an error happened during the process
error_happened = False

try:
    # Set the command to run when launching the container, by overriding a
    # constant.
    # Task constants are the main way of controlling a task's behaviour
    task.constants['DOCKER_REPO'] = 'microsoft/dotnet'
    task.constants['DOCKER_TAG'] = 'latest'
    task.constants['DOCKER_CMD'] = dotnet_cmd

    # Create a resource disk and add our input file.
    input_disk = conn.create_disk('processing-image-resource')
    input_disk.add_directory("bin")
    input_disk.add_file("test.jpg")

    # Attach the disk to the task
    task.resources.append(input_disk)

    # Submit the task to the Api, that will launch it on the cluster
    task.submit()

    # Wait for the task to be finished, and monitor the progress of its
    # deployment
    last_state = ''
    done = False
    while not done:
        if task.state != last_state:
            last_state = task.state
            print("** {}".format(last_state))

        # Wait for the task to complete, with a timeout of 5 seconds.
        # This will return True as soon as the task is complete, or False
        # after the timeout.
        done = task.wait(60)

        # Display fresh stdout / stderr
        sys.stdout.write(task.fresh_stdout())
        sys.stderr.write(task.fresh_stderr())

    if task.state == 'Failure':
        # Display errors on failure
        print("** Errors: %s" % task.errors[0])
        error_happened = True
    else:
        # Or download the results
        task.download_results('.')

finally:
    task.delete(purge_resources=True, purge_results=True)
    # Exit code in case of error
    if error_happened:
        sys.exit(1)

Well, few things to explain :

# Create a connection, from which all other objects will be derived
conn = qarnot.Connection('qarnot.conf')

You need to create the ‘qarnot.conf’ file to store your Token API.

[cluster]
# url of the REST API
url=https://api.qarnot.com
[client]
# auth string of the client
token=__YOUR_TOKEN_API__

Here is my folders structure, where bin contains the .NET Core console app and qarnot the pyhon SDK. And yes, test.jpg is the image you want to process ;)

It’s time to execute our script and show the result !

$> python test.py
** Submitted
** FullyDispatched
 0> Tue Feb 27 09:02:48 CET 2018
$>

Qarnot provide a monitoring portal that allow you to show all your tasks and disks.

At the end of the execution, you now have a new image named new_test.jpg in your root folder.

 

 

 

 

 

 

 

Sources : https://github.com/Lordinaire/Qarnot-NetCore

Working with Docker in Visual Studio 2017

Docker is a well-know tool in Microsoft ecosystem since few years. You can of course doing great things if you deal with Azure but it isn’t limited to “ops”. Developers can now easily play with Docker inside Visual Studio with some additional tools. Stuff like starting a container becomes so easy that everybody must use it! (I like Docker, a lot!)

Setup & Tools

First, you need to download and install Docker for Windows and select the .NET Core option in the installer of Visual Studio 2017.

Bonus: You can additionally download Kitematic (Visual Docker Container Management) if the CLI is not your friend.

Project sample

Let’s create a new ASP.NET Core 2.0 project of type WebAPI. The main benefit to target a .NET Core project is that it can run on Windows AND Linux platform.

Docker for Windows is able to run both of platforms type for a container but not a mixed use of it. So, you need that all your containers target the same platform, you can’t set a specific platform for a container. That’s the only limitation that I face off (and it can be a huge one…).

You can check the Enable Docker Support now or add it after when the solution will be created. If you doesn’t set the option, le solution explorer looks like that :

To activate the Docker support, right clic on the project and select Add > Enable Docker Support. Visual Studio will now automatically create a new project named docker-compose and a Dockerfile in your project. Well, thank you VS.

Dockerfile

FROM microsoft/aspnetcore:2.0
ARG source
WORKDIR /app
EXPOSE 80
COPY ${source:-obj/Docker/publish} .
ENTRYPOINT ["dotnet", "WebApplication1.dll"]

docker-compose.yml

version: '3'

services:
  webapplication1:
    image: webapplication1
  build:
    context: ./WebApplication1
    dockerfile: Dockerfile

Start the project

Before starting the project, we need to check if the shared drives settings are set properly because Docker required full access to drives where the source code.

Update 04/11/17 : This is only needed if you use Linux containers with Docker for Windows. You can avoid that by using the option Switching to Windows containers.

 

To run your project with Docker, you need to set the docker-compose project as startup project. Hit F5 and open the console window : you will see all the steps like downloading the image, creating the container, …

Bonus: You can add breakpoints like you always do and it will work!

Bonus 2: If you launch Kitematic you will see your container with all his settings

 

 

 

Page 1 of 2

Powered by WordPress & Theme by Anders Norén

%d bloggers like this: