Управление вычислениями

بواسطة: Stepik

Overview

Технические аспекты анализа данных зачастую влияют на качество получаемых результатов. В последние годы в центре внимания находятся вопросы обеспечения воспроизводимости и автоматизации анализа – попытки воспроизвести результаты, полученные той или иной группой, часто проваливаются, а анализ больших массивов данных нередко проводится с использованием подручных средств.

Курс состоит из трех модулей, в которых рассматриваются Linux контейнеры и Docker (первый модуль), а также различные подходы к автоматизации процесса вычислений и инструменты, реализующие эти подходы (второй и третий модули). Использование этих инструментов позволяет сделать процесс анализа более воспроизводимым и эффективным.

 

Syllabus

Введение


1.1 Введение

Контейнеризация и Docker


2.1 Знакомство с Docker


2.2 Работа с файлами в контейнерах Docker


2.3 Взаимодействие контейнеров Docker


2.4 Работа с образами Docker


2.5 Технологические основы и роль контейнеризации

Workflows и Pipelines


3.1 Работа с данными и пайплайны


3.2 Инструменты пайплайнинга и подходы к описанию пайплайнов


3.3 Snakemake: имплицитное описание графа анализа


3.4 Snakemake: создание пайплайнов

Common Workflow Language


4.1 Знакомство с CWL


4.2 Описание инструментов с использованием CWL


4.3 CWL: полезные компоненты


4.4 CWL: создание пайплайнов

Заключение


5.1 Заключение

Taught by

Anton Bragin and Sergei Mozgov

Управление вычислениями
الذهاب الي الدورة

Управление вычислениями

بواسطة: Stepik

  • Stepik
  • مجانية
  • Russian
  • متاح شهادة
  • متاح في أي وقت
  • الجميع
  • N/A
8.1.2PHP Version319msRequest Duration2MBMemory UsageGET ar/الدورات/{slug}Route
    • Booting (202ms)
    • Application (116ms)
    • 1 x Booting (63.32%)
      202.27ms
      1 x Application (36.43%)
      116.39ms
      14 templates were rendered
      • public.courses.show (resources/views/public/courses/show.blade.php)3bladefile
        Params
        0
        course
        1
        links
        2
        config
      • public.courses.partials.breadcrumbs (resources/views/public/courses/partials/breadcrumbs.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.heading (resources/views/public/courses/partials/heading.blade.php)7bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        classes
      • public.courses.partials.details (resources/views/public/courses/partials/details.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.breadcrumbs (resources/views/public/courses/partials/breadcrumbs.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.heading (resources/views/public/courses/partials/heading.blade.php)7bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        classes
      • public.layouts.main (resources/views/public/layouts/main.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.layouts.partials.meta (resources/views/public/layouts/partials/meta.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.layouts.partials.navbar (resources/views/public/layouts/partials/navbar.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.auth.profile.partials.links (resources/views/public/auth/profile/partials/links.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.layouts.partials.flash-session (resources/views/public/layouts/partials/flash-session.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      uri
      GET ar/الدورات/{slug}
      middleware
      web, localize:ar
      controller
      App\Http\Controllers\CourseController@show
      as
      ar.courses.show
      namespace
      prefix
      /ar
      where
      file
      app/Http/Controllers/CourseController.php:17-35
      6 statements were executed6.33ms
      • select * from `courses` where `slug_ar` = 'управление-вычислениями' limit 1
        4.97ms/app/Http/Controllers/CourseController.php:20corspedia
        Metadata
        Bindings
        • 0. управление-вычислениями
        Backtrace
        • 17. /app/Http/Controllers/CourseController.php:20
        • 18. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 19. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 20. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • update `courses` set `visitors` = `visitors` + 1, `courses`.`updated_at` = '2025-06-08 06:15:31' where `id` = 1927
        680μs/app/Http/Controllers/CourseController.php:21corspedia
        Metadata
        Bindings
        • 0. 2025-06-08 06:15:31
        • 1. 1927
        Backtrace
        • 17. /app/Http/Controllers/CourseController.php:21
        • 18. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 19. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 20. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select `id`, `name_en`, `name_ar`, `topic_id`, `slug_en`, `slug_ar` from `subjects` where `subjects`.`id` in (10)
        170μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select `id`, `name_en`, `name_ar`, `slug_en`, `slug_ar` from `topics` where `topics`.`id` in (1)
        140μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 25. /app/Http/Controllers/CourseController.php:23
        • 26. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 27. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 28. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 29. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `providers` where `providers`.`id` in (33) and `providers`.`deleted_at` is null
        170μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `html_files` where `html_files`.`id` = 1918 limit 1
        200μs/app/Models/Course.php:84corspedia
        Metadata
        Bindings
        • 0. 1918
        Backtrace
        • 21. /app/Models/Course.php:84
        • 28. view::public.courses.show:29
        • 30. /vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:125
        • 31. /vendor/laravel/framework/src/Illuminate/View/Engines/PhpEngine.php:58
        • 32. /vendor/laravel/framework/src/Illuminate/View/Engines/CompilerEngine.php:72
      App\Models\HtmlFile
      1
      App\Models\Provider
      1
      App\Models\Topic
      1
      App\Models\Subject
      1
      App\Models\Course
      1
        _token
        SPzZtBDQQSk0nDp68F1ZopzGpbZLxmwDZZ8u4iEN
        locale
        ar
        _previous
        array:1 [ "url" => "https://www.corspedia.com/ar/%D8%A7%D9%84%D8%AF%D9%88%D8%B1%D8%A7%D8%AA/%D1%83...
        _flash
        array:2 [ "old" => [] "new" => [] ]
        PHPDEBUGBAR_STACK_DATA
        []
        path_info
        /ar/%D8%A7%D9%84%D8%AF%D9%88%D8%B1%D8%A7%D8%AA/%D1%83%D0%BF%D1%80%D0%B0%D0%B2%D0%BB%D0%B5%D0%BD%D0%B8%D0%B5-%D0%B2%D1%8B%D1%87%D0%B8%D1%81%D0%BB%D0%B5%D0%BD%D0%B8%D1%8F%D0%BC%D0%B8
        status_code
        200
        
        status_text
        OK
        format
        html
        content_type
        text/html; charset=UTF-8
        request_query
        []
        
        request_request
        []
        
        request_headers
        0 of 0
        array:24 [ "cf-ipcountry" => array:1 [ 0 => "US" ] "cf-connecting-ip" => array:1 [ 0 => "216.73.216.18" ] "cdn-loop" => array:1 [ 0 => "cloudflare; loops=1" ] "x-forwarded-proto" => array:1 [ 0 => "https" ] "x-forwarded-for" => array:1 [ 0 => "216.73.216.18" ] "sec-fetch-site" => array:1 [ 0 => "none" ] "accept" => array:1 [ 0 => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" ] "user-agent" => array:1 [ 0 => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" ] "upgrade-insecure-requests" => array:1 [ 0 => "1" ] "sec-ch-ua-platform" => array:1 [ 0 => ""Windows"" ] "sec-ch-ua-mobile" => array:1 [ 0 => "?0" ] "sec-ch-ua" => array:1 [ 0 => ""Chromium";v="130", "HeadlessChrome";v="130", "Not?A_Brand";v="99"" ] "cache-control" => array:1 [ 0 => "no-cache" ] "pragma" => array:1 [ 0 => "no-cache" ] "sec-fetch-dest" => array:1 [ 0 => "document" ] "cf-ray" => array:1 [ 0 => "94c641550dd8d130-ORD" ] "accept-encoding" => array:1 [ 0 => "gzip, br" ] "priority" => array:1 [ 0 => "u=0, i" ] "sec-fetch-user" => array:1 [ 0 => "?1" ] "sec-fetch-mode" => array:1 [ 0 => "navigate" ] "cf-visitor" => array:1 [ 0 => "{"scheme":"https"}" ] "host" => array:1 [ 0 => "www.corspedia.com" ] "content-length" => array:1 [ 0 => "" ] "content-type" => array:1 [ 0 => "" ] ]
        request_server
        0 of 0
        array:50 [ "USER" => "www-data" "HOME" => "/var/www" "HTTP_CF_IPCOUNTRY" => "US" "HTTP_CF_CONNECTING_IP" => "216.73.216.18" "HTTP_CDN_LOOP" => "cloudflare; loops=1" "HTTP_X_FORWARDED_PROTO" => "https" "HTTP_X_FORWARDED_FOR" => "216.73.216.18" "HTTP_SEC_FETCH_SITE" => "none" "HTTP_ACCEPT" => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" "HTTP_USER_AGENT" => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" "HTTP_UPGRADE_INSECURE_REQUESTS" => "1" "HTTP_SEC_CH_UA_PLATFORM" => ""Windows"" "HTTP_SEC_CH_UA_MOBILE" => "?0" "HTTP_SEC_CH_UA" => ""Chromium";v="130", "HeadlessChrome";v="130", "Not?A_Brand";v="99"" "HTTP_CACHE_CONTROL" => "no-cache" "HTTP_PRAGMA" => "no-cache" "HTTP_SEC_FETCH_DEST" => "document" "HTTP_CF_RAY" => "94c641550dd8d130-ORD" "HTTP_ACCEPT_ENCODING" => "gzip, br" "HTTP_PRIORITY" => "u=0, i" "HTTP_SEC_FETCH_USER" => "?1" "HTTP_SEC_FETCH_MODE" => "navigate" "HTTP_CF_VISITOR" => "{"scheme":"https"}" "HTTP_HOST" => "www.corspedia.com" "REDIRECT_STATUS" => "200" "SERVER_NAME" => "corspedia.com" "SERVER_PORT" => "443" "SERVER_ADDR" => "141.95.147.152" "REMOTE_USER" => "" "REMOTE_PORT" => "46024" "REMOTE_ADDR" => "172.70.194.133" "SERVER_SOFTWARE" => "nginx/1.18.0" "GATEWAY_INTERFACE" => "CGI/1.1" "HTTPS" => "on" "REQUEST_SCHEME" => "https" "SERVER_PROTOCOL" => "HTTP/2.0" "DOCUMENT_ROOT" => "/var/www/corspedia/public" "DOCUMENT_URI" => "/index.php" "REQUEST_URI" => "/ar/%D8%A7%D9%84%D8%AF%D9%88%D8%B1%D8%A7%D8%AA/%D1%83%D0%BF%D1%80%D0%B0%D0%B2%D0%BB%D0%B5%D0%BD%D0%B8%D0%B5-%D0%B2%D1%8B%D1%87%D0%B8%D1%81%D0%BB%D0%B5%D0%BD%D0%B8%D1%8F%D0%BC%D0%B8/ar/%D8%A7%D9%84%D8%AF%D9%88%D8%B1%D8%A7%D8%AA/%D1%83%D0%BF%D1%80%D0%B0%D0%B2%D0%BB%D0%B5%D0%BD%D0%B8%D0%B5-%D0%B2%D1%8B%D1%87%D0%B8%D1%81%D0%BB%D0%B5%D0%BD%D0%" "SCRIPT_NAME" => "/index.php" "CONTENT_LENGTH" => "" "CONTENT_TYPE" => "" "REQUEST_METHOD" => "GET" "QUERY_STRING" => "" "SCRIPT_FILENAME" => "/var/www/corspedia/public/index.php" "PATH_INFO" => "" "FCGI_ROLE" => "RESPONDER" "PHP_SELF" => "/index.php" "REQUEST_TIME_FLOAT" => 1749363331.621 "REQUEST_TIME" => 1749363331 ]
        request_cookies
        []
        
        response_headers
        0 of 0
        array:5 [ "content-type" => array:1 [ 0 => "text/html; charset=UTF-8" ] "cache-control" => array:1 [ 0 => "no-cache, private" ] "date" => array:1 [ 0 => "Sun, 08 Jun 2025 06:15:31 GMT" ] "set-cookie" => array:2 [ 0 => "XSRF-TOKEN=eyJpdiI6Ikx2M2RZK0crUzE4c0xSNHVzUyszMkE9PSIsInZhbHVlIjoiNXd5YStVSmtNM1I1WVE5VjhkaHpzK01xTTBDVmZMZ3hxemtnZkpTSlM4SmlZL1RMeHFSV25CdWIvMjdTZVgyK2J0M25NY1Y3TTdMUHhPUkg3WTNIUTEySUdPNmM2K1hxUFRiRGZhMGdHMnVZZ0p3ZExVcml5ZnkyZVY5MVFOYVYiLCJtYWMiOiJkZDkyYzA4MGVlNmRiNmI5YjNhM2E1ZWM0ODQ3NWZlOGM0YzU4M2ZkNTRhMjdmMzVlYTAxZjZmNTIzZjYwNjkzIiwidGFnIjoiIn0%3D; expires=Sun, 08 Jun 2025 08:15:31 GMT; Max-Age=7200; path=/; samesite=laxXSRF-TOKEN=eyJpdiI6Ikx2M2RZK0crUzE4c0xSNHVzUyszMkE9PSIsInZhbHVlIjoiNXd5YStVSmtNM1I1WVE5VjhkaHpzK01xTTBDVmZMZ3hxemtnZkpTSlM4SmlZL1RMeHFSV25CdWIvMjdTZVgyK2J0M25NY" 1 => "laravel_session=eyJpdiI6Ikg4SS9xRzZMcWN6OVBvdkI0TU1Xa1E9PSIsInZhbHVlIjoiS3NzZVVSUUhFY0orN3BlQmdyUnVXWmRYZHJrcGhDMkZzTHR0ekVWRlQyekJyODVXa1lDczduR0J1a2l0MDZ2dmZEc1JSSW9LMTU5dUtBdTZBYlBrMDJRT3pOSlR1RXJsWXZtTGJhdCtXVnZDaExHUWVhaFJjMzcweC9hckhoUkIiLCJtYWMiOiIyYmU4ZDQ3ZWNkZWU0NTdhNWNjNTNmMDdhOTA1NWYxZTQ4NmNlNGRkZjRlZTQ2OWIzYjBjNjI3OWE5OWIwOWQ2IiwidGFnIjoiIn0%3D; expires=Sun, 08 Jun 2025 08:15:31 GMT; Max-Age=7200; path=/; httponly; samesite=laxlaravel_session=eyJpdiI6Ikg4SS9xRzZMcWN6OVBvdkI0TU1Xa1E9PSIsInZhbHVlIjoiS3NzZVVSUUhFY0orN3BlQmdyUnVXWmRYZHJrcGhDMkZzTHR0ekVWRlQyekJyODVXa1lDczduR0J1a2l0MDZ2dmZE" ] "Set-Cookie" => array:2 [ 0 => "XSRF-TOKEN=eyJpdiI6Ikx2M2RZK0crUzE4c0xSNHVzUyszMkE9PSIsInZhbHVlIjoiNXd5YStVSmtNM1I1WVE5VjhkaHpzK01xTTBDVmZMZ3hxemtnZkpTSlM4SmlZL1RMeHFSV25CdWIvMjdTZVgyK2J0M25NY1Y3TTdMUHhPUkg3WTNIUTEySUdPNmM2K1hxUFRiRGZhMGdHMnVZZ0p3ZExVcml5ZnkyZVY5MVFOYVYiLCJtYWMiOiJkZDkyYzA4MGVlNmRiNmI5YjNhM2E1ZWM0ODQ3NWZlOGM0YzU4M2ZkNTRhMjdmMzVlYTAxZjZmNTIzZjYwNjkzIiwidGFnIjoiIn0%3D; expires=Sun, 08-Jun-2025 08:15:31 GMT; path=/XSRF-TOKEN=eyJpdiI6Ikx2M2RZK0crUzE4c0xSNHVzUyszMkE9PSIsInZhbHVlIjoiNXd5YStVSmtNM1I1WVE5VjhkaHpzK01xTTBDVmZMZ3hxemtnZkpTSlM4SmlZL1RMeHFSV25CdWIvMjdTZVgyK2J0M25NY" 1 => "laravel_session=eyJpdiI6Ikg4SS9xRzZMcWN6OVBvdkI0TU1Xa1E9PSIsInZhbHVlIjoiS3NzZVVSUUhFY0orN3BlQmdyUnVXWmRYZHJrcGhDMkZzTHR0ekVWRlQyekJyODVXa1lDczduR0J1a2l0MDZ2dmZEc1JSSW9LMTU5dUtBdTZBYlBrMDJRT3pOSlR1RXJsWXZtTGJhdCtXVnZDaExHUWVhaFJjMzcweC9hckhoUkIiLCJtYWMiOiIyYmU4ZDQ3ZWNkZWU0NTdhNWNjNTNmMDdhOTA1NWYxZTQ4NmNlNGRkZjRlZTQ2OWIzYjBjNjI3OWE5OWIwOWQ2IiwidGFnIjoiIn0%3D; expires=Sun, 08-Jun-2025 08:15:31 GMT; path=/; httponlylaravel_session=eyJpdiI6Ikg4SS9xRzZMcWN6OVBvdkI0TU1Xa1E9PSIsInZhbHVlIjoiS3NzZVVSUUhFY0orN3BlQmdyUnVXWmRYZHJrcGhDMkZzTHR0ekVWRlQyekJyODVXa1lDczduR0J1a2l0MDZ2dmZE" ] ]
        session_attributes
        0 of 0
        array:5 [ "_token" => "SPzZtBDQQSk0nDp68F1ZopzGpbZLxmwDZZ8u4iEN" "locale" => "ar" "_previous" => array:1 [ "url" => "https://www.corspedia.com/ar/%D8%A7%D9%84%D8%AF%D9%88%D8%B1%D8%A7%D8%AA/%D1%83%D0%BF%D1%80%D0%B0%D0%B2%D0%BB%D0%B5%D0%BD%D0%B8%D0%B5-%D0%B2%D1%8B%D1%87%D0%B8%D1%81%D0%BB%D0%B5%D0%BD%D0%B8%D1%8F%D0%BC%D0%B8https://www.corspedia.com/ar/%D8%A7%D9%84%D8%AF%D9%88%D8%B1%D8%A7%D8%AA/%D1%83%D0%BF%D1%80%D0%B0%D0%B2%D0%BB%D0%B5%D0%BD%D0%B8%D0%B5-%D0%B2%D1%8B%D1%87%D0%B8%D1" ] "_flash" => array:2 [ "old" => [] "new" => [] ] "PHPDEBUGBAR_STACK_DATA" => [] ]