Управление вычислениями

Brought by: Stepik

Overview

Технические аспекты анализа данных зачастую влияют на качество получаемых результатов. В последние годы в центре внимания находятся вопросы обеспечения воспроизводимости и автоматизации анализа – попытки воспроизвести результаты, полученные той или иной группой, часто проваливаются, а анализ больших массивов данных нередко проводится с использованием подручных средств.

Курс состоит из трех модулей, в которых рассматриваются Linux контейнеры и Docker (первый модуль), а также различные подходы к автоматизации процесса вычислений и инструменты, реализующие эти подходы (второй и третий модули). Использование этих инструментов позволяет сделать процесс анализа более воспроизводимым и эффективным.

 

Syllabus

Введение


1.1 Введение

Контейнеризация и Docker


2.1 Знакомство с Docker


2.2 Работа с файлами в контейнерах Docker


2.3 Взаимодействие контейнеров Docker


2.4 Работа с образами Docker


2.5 Технологические основы и роль контейнеризации

Workflows и Pipelines


3.1 Работа с данными и пайплайны


3.2 Инструменты пайплайнинга и подходы к описанию пайплайнов


3.3 Snakemake: имплицитное описание графа анализа


3.4 Snakemake: создание пайплайнов

Common Workflow Language


4.1 Знакомство с CWL


4.2 Описание инструментов с использованием CWL


4.3 CWL: полезные компоненты


4.4 CWL: создание пайплайнов

Заключение


5.1 Заключение

Taught by

Anton Bragin and Sergei Mozgov

Управление вычислениями
Go to course

Управление вычислениями

Brought by: Stepik

  • Stepik
  • Free
  • Russian
  • Certificate Available
  • Available at any time
  • All
  • N/A
8.1.2PHP Version1.11sRequest Duration2MBMemory UsageGET en/courses/{slug}Route
    • Booting (720ms)
    • Application (386ms)
    • 1 x Booting (64.91%)
      719.53ms
      1 x Application (34.84%)
      386.23ms
      14 templates were rendered
      • public.courses.show (resources/views/public/courses/show.blade.php)3bladefile
        Params
        0
        course
        1
        links
        2
        config
      • public.courses.partials.breadcrumbs (resources/views/public/courses/partials/breadcrumbs.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.heading (resources/views/public/courses/partials/heading.blade.php)7bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        classes
      • public.courses.partials.details (resources/views/public/courses/partials/details.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.breadcrumbs (resources/views/public/courses/partials/breadcrumbs.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.heading (resources/views/public/courses/partials/heading.blade.php)7bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        classes
      • public.layouts.main (resources/views/public/layouts/main.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.layouts.partials.meta (resources/views/public/layouts/partials/meta.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.layouts.partials.navbar (resources/views/public/layouts/partials/navbar.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.auth.profile.partials.links (resources/views/public/auth/profile/partials/links.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.layouts.partials.flash-session (resources/views/public/layouts/partials/flash-session.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      uri
      GET en/courses/{slug}
      middleware
      web, localize:en
      controller
      App\Http\Controllers\CourseController@show
      as
      en.courses.show
      namespace
      prefix
      /en
      where
      file
      app/Http/Controllers/CourseController.php:17-35
      6 statements were executed15.26ms
      • select * from `courses` where `slug_en` = 'управление-вычислениями' limit 1
        8.88ms/app/Http/Controllers/CourseController.php:20corspedia
        Metadata
        Bindings
        • 0. управление-вычислениями
        Backtrace
        • 17. /app/Http/Controllers/CourseController.php:20
        • 18. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 19. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 20. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • update `courses` set `visitors` = `visitors` + 1, `courses`.`updated_at` = '2025-06-22 05:08:23' where `id` = 1927
        5.59ms/app/Http/Controllers/CourseController.php:21corspedia
        Metadata
        Bindings
        • 0. 2025-06-22 05:08:23
        • 1. 1927
        Backtrace
        • 17. /app/Http/Controllers/CourseController.php:21
        • 18. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 19. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 20. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select `id`, `name_en`, `name_ar`, `topic_id`, `slug_en`, `slug_ar` from `subjects` where `subjects`.`id` in (10)
        200μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select `id`, `name_en`, `name_ar`, `slug_en`, `slug_ar` from `topics` where `topics`.`id` in (1)
        160μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 25. /app/Http/Controllers/CourseController.php:23
        • 26. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 27. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 28. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 29. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `providers` where `providers`.`id` in (33) and `providers`.`deleted_at` is null
        180μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `html_files` where `html_files`.`id` = 1918 limit 1
        250μs/app/Models/Course.php:84corspedia
        Metadata
        Bindings
        • 0. 1918
        Backtrace
        • 21. /app/Models/Course.php:84
        • 28. view::public.courses.show:29
        • 30. /vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:125
        • 31. /vendor/laravel/framework/src/Illuminate/View/Engines/PhpEngine.php:58
        • 32. /vendor/laravel/framework/src/Illuminate/View/Engines/CompilerEngine.php:72
      App\Models\HtmlFile
      1
      App\Models\Provider
      1
      App\Models\Topic
      1
      App\Models\Subject
      1
      App\Models\Course
      1
        _token
        WHlut5sq8JIUley8NEW24pmyHM0HJeCQypqQiAvE
        locale
        en
        _previous
        array:1 [ "url" => "https://www.corspedia.com/en/courses/%D1%83%D0%BF%D1%80%D0%B0%D0%B2%D0%BB%D0%B...
        _flash
        array:2 [ "old" => [] "new" => [] ]
        PHPDEBUGBAR_STACK_DATA
        []
        path_info
        /en/courses/%D1%83%D0%BF%D1%80%D0%B0%D0%B2%D0%BB%D0%B5%D0%BD%D0%B8%D0%B5-%D0%B2%D1%8B%D1%87%D0%B8%D1%81%D0%BB%D0%B5%D0%BD%D0%B8%D1%8F%D0%BC%D0%B8
        status_code
        200
        
        status_text
        OK
        format
        html
        content_type
        text/html; charset=UTF-8
        request_query
        []
        
        request_request
        []
        
        request_headers
        0 of 0
        array:24 [ "cf-ipcountry" => array:1 [ 0 => "US" ] "cf-connecting-ip" => array:1 [ 0 => "216.73.216.27" ] "cdn-loop" => array:1 [ 0 => "cloudflare; loops=1" ] "x-forwarded-proto" => array:1 [ 0 => "https" ] "x-forwarded-for" => array:1 [ 0 => "216.73.216.27" ] "sec-fetch-site" => array:1 [ 0 => "none" ] "accept" => array:1 [ 0 => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" ] "user-agent" => array:1 [ 0 => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" ] "upgrade-insecure-requests" => array:1 [ 0 => "1" ] "sec-ch-ua-platform" => array:1 [ 0 => ""Windows"" ] "sec-ch-ua-mobile" => array:1 [ 0 => "?0" ] "sec-ch-ua" => array:1 [ 0 => ""Chromium";v="130", "HeadlessChrome";v="130", "Not?A_Brand";v="99"" ] "cache-control" => array:1 [ 0 => "no-cache" ] "pragma" => array:1 [ 0 => "no-cache" ] "sec-fetch-dest" => array:1 [ 0 => "document" ] "cf-ray" => array:1 [ 0 => "95393a35fcb40042-ORD" ] "accept-encoding" => array:1 [ 0 => "gzip, br" ] "priority" => array:1 [ 0 => "u=0, i" ] "sec-fetch-user" => array:1 [ 0 => "?1" ] "sec-fetch-mode" => array:1 [ 0 => "navigate" ] "cf-visitor" => array:1 [ 0 => "{"scheme":"https"}" ] "host" => array:1 [ 0 => "www.corspedia.com" ] "content-length" => array:1 [ 0 => "" ] "content-type" => array:1 [ 0 => "" ] ]
        request_server
        0 of 0
        array:50 [ "USER" => "www-data" "HOME" => "/var/www" "HTTP_CF_IPCOUNTRY" => "US" "HTTP_CF_CONNECTING_IP" => "216.73.216.27" "HTTP_CDN_LOOP" => "cloudflare; loops=1" "HTTP_X_FORWARDED_PROTO" => "https" "HTTP_X_FORWARDED_FOR" => "216.73.216.27" "HTTP_SEC_FETCH_SITE" => "none" "HTTP_ACCEPT" => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" "HTTP_USER_AGENT" => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" "HTTP_UPGRADE_INSECURE_REQUESTS" => "1" "HTTP_SEC_CH_UA_PLATFORM" => ""Windows"" "HTTP_SEC_CH_UA_MOBILE" => "?0" "HTTP_SEC_CH_UA" => ""Chromium";v="130", "HeadlessChrome";v="130", "Not?A_Brand";v="99"" "HTTP_CACHE_CONTROL" => "no-cache" "HTTP_PRAGMA" => "no-cache" "HTTP_SEC_FETCH_DEST" => "document" "HTTP_CF_RAY" => "95393a35fcb40042-ORD" "HTTP_ACCEPT_ENCODING" => "gzip, br" "HTTP_PRIORITY" => "u=0, i" "HTTP_SEC_FETCH_USER" => "?1" "HTTP_SEC_FETCH_MODE" => "navigate" "HTTP_CF_VISITOR" => "{"scheme":"https"}" "HTTP_HOST" => "www.corspedia.com" "REDIRECT_STATUS" => "200" "SERVER_NAME" => "corspedia.com" "SERVER_PORT" => "443" "SERVER_ADDR" => "141.95.147.152" "REMOTE_USER" => "" "REMOTE_PORT" => "54188" "REMOTE_ADDR" => "172.70.127.102" "SERVER_SOFTWARE" => "nginx/1.18.0" "GATEWAY_INTERFACE" => "CGI/1.1" "HTTPS" => "on" "REQUEST_SCHEME" => "https" "SERVER_PROTOCOL" => "HTTP/2.0" "DOCUMENT_ROOT" => "/var/www/corspedia/public" "DOCUMENT_URI" => "/index.php" "REQUEST_URI" => "/en/courses/%D1%83%D0%BF%D1%80%D0%B0%D0%B2%D0%BB%D0%B5%D0%BD%D0%B8%D0%B5-%D0%B2%D1%8B%D1%87%D0%B8%D1%81%D0%BB%D0%B5%D0%BD%D0%B8%D1%8F%D0%BC%D0%B8" "SCRIPT_NAME" => "/index.php" "CONTENT_LENGTH" => "" "CONTENT_TYPE" => "" "REQUEST_METHOD" => "GET" "QUERY_STRING" => "" "SCRIPT_FILENAME" => "/var/www/corspedia/public/index.php" "PATH_INFO" => "" "FCGI_ROLE" => "RESPONDER" "PHP_SELF" => "/index.php" "REQUEST_TIME_FLOAT" => 1750568902.335 "REQUEST_TIME" => 1750568902 ]
        request_cookies
        []
        
        response_headers
        0 of 0
        array:5 [ "content-type" => array:1 [ 0 => "text/html; charset=UTF-8" ] "cache-control" => array:1 [ 0 => "no-cache, private" ] "date" => array:1 [ 0 => "Sun, 22 Jun 2025 05:08:23 GMT" ] "set-cookie" => array:2 [ 0 => "XSRF-TOKEN=eyJpdiI6Ijc5RExWNU1WNnFrNk1lQ2F3Y1lOclE9PSIsInZhbHVlIjoiblVPNGNOQnZhOUp6enIxdm0ybTlWa0h5bExRZWlQQ0JDbDBmWEtiTDRpWUhzR29LS2FDTkM2Nlhvc2pjUDlROW15YmVpclMwTFB2SW1saFJOY3J5L2tKcU9wcjJvUEFQeXZCb2xFeEplQ1pEUTNsaFZIZFNCUGkyZWJ1aFdCT3oiLCJtYWMiOiI2MmNiOGE3ZGFjOWJjM2QzNmRlZjljYTM0MGZhZGE4OGJkMGE4Y2RkMzFjYmEzNTI5NmM1YjdmZDgyYjcwMWRhIiwidGFnIjoiIn0%3D; expires=Sun, 22 Jun 2025 07:08:23 GMT; Max-Age=7200; path=/; samesite=laxXSRF-TOKEN=eyJpdiI6Ijc5RExWNU1WNnFrNk1lQ2F3Y1lOclE9PSIsInZhbHVlIjoiblVPNGNOQnZhOUp6enIxdm0ybTlWa0h5bExRZWlQQ0JDbDBmWEtiTDRpWUhzR29LS2FDTkM2Nlhvc2pjUDlROW15YmVpc" 1 => "laravel_session=eyJpdiI6IkNjMUUzMGszaEMvd25BTjloNjhoZEE9PSIsInZhbHVlIjoieDFVdVdCTU1XdXcxVW9LNXU0R3FuWnRsU2NRb0lsUVgxVXNXNy9XckpjK1diUXZ0SzFhcFpZT240TmdPU3NFSmJGaUY5d0Z3cmQrV001ekxTZzEvRHlqRDRuUXhsYk81VEdSM1FYS295ZEF5aThvMWlYUmNoSlpoTmF2aTJrL1UiLCJtYWMiOiIyOTNkZDU3OTdkZmU3ODM4NWQxYWVkYjg5OWEwMjViZTQzNjk2YTk2ZGJlMzEyM2ZlZTAyMDM5MTg0ZGZlMzU2IiwidGFnIjoiIn0%3D; expires=Sun, 22 Jun 2025 07:08:23 GMT; Max-Age=7200; path=/; httponly; samesite=laxlaravel_session=eyJpdiI6IkNjMUUzMGszaEMvd25BTjloNjhoZEE9PSIsInZhbHVlIjoieDFVdVdCTU1XdXcxVW9LNXU0R3FuWnRsU2NRb0lsUVgxVXNXNy9XckpjK1diUXZ0SzFhcFpZT240TmdPU3NFSmJG" ] "Set-Cookie" => array:2 [ 0 => "XSRF-TOKEN=eyJpdiI6Ijc5RExWNU1WNnFrNk1lQ2F3Y1lOclE9PSIsInZhbHVlIjoiblVPNGNOQnZhOUp6enIxdm0ybTlWa0h5bExRZWlQQ0JDbDBmWEtiTDRpWUhzR29LS2FDTkM2Nlhvc2pjUDlROW15YmVpclMwTFB2SW1saFJOY3J5L2tKcU9wcjJvUEFQeXZCb2xFeEplQ1pEUTNsaFZIZFNCUGkyZWJ1aFdCT3oiLCJtYWMiOiI2MmNiOGE3ZGFjOWJjM2QzNmRlZjljYTM0MGZhZGE4OGJkMGE4Y2RkMzFjYmEzNTI5NmM1YjdmZDgyYjcwMWRhIiwidGFnIjoiIn0%3D; expires=Sun, 22-Jun-2025 07:08:23 GMT; path=/XSRF-TOKEN=eyJpdiI6Ijc5RExWNU1WNnFrNk1lQ2F3Y1lOclE9PSIsInZhbHVlIjoiblVPNGNOQnZhOUp6enIxdm0ybTlWa0h5bExRZWlQQ0JDbDBmWEtiTDRpWUhzR29LS2FDTkM2Nlhvc2pjUDlROW15YmVpc" 1 => "laravel_session=eyJpdiI6IkNjMUUzMGszaEMvd25BTjloNjhoZEE9PSIsInZhbHVlIjoieDFVdVdCTU1XdXcxVW9LNXU0R3FuWnRsU2NRb0lsUVgxVXNXNy9XckpjK1diUXZ0SzFhcFpZT240TmdPU3NFSmJGaUY5d0Z3cmQrV001ekxTZzEvRHlqRDRuUXhsYk81VEdSM1FYS295ZEF5aThvMWlYUmNoSlpoTmF2aTJrL1UiLCJtYWMiOiIyOTNkZDU3OTdkZmU3ODM4NWQxYWVkYjg5OWEwMjViZTQzNjk2YTk2ZGJlMzEyM2ZlZTAyMDM5MTg0ZGZlMzU2IiwidGFnIjoiIn0%3D; expires=Sun, 22-Jun-2025 07:08:23 GMT; path=/; httponlylaravel_session=eyJpdiI6IkNjMUUzMGszaEMvd25BTjloNjhoZEE9PSIsInZhbHVlIjoieDFVdVdCTU1XdXcxVW9LNXU0R3FuWnRsU2NRb0lsUVgxVXNXNy9XckpjK1diUXZ0SzFhcFpZT240TmdPU3NFSmJG" ] ]
        session_attributes
        0 of 0
        array:5 [ "_token" => "WHlut5sq8JIUley8NEW24pmyHM0HJeCQypqQiAvE" "locale" => "en" "_previous" => array:1 [ "url" => "https://www.corspedia.com/en/courses/%D1%83%D0%BF%D1%80%D0%B0%D0%B2%D0%BB%D0%B5%D0%BD%D0%B8%D0%B5-%D0%B2%D1%8B%D1%87%D0%B8%D1%81%D0%BB%D0%B5%D0%BD%D0%B8%D1%8F%D0%BC%D0%B8https://www.corspedia.com/en/courses/%D1%83%D0%BF%D1%80%D0%B0%D0%B2%D0%BB%D0%B5%D0%BD%D0%B8%D0%B5-%D0%B2%D1%8B%D1%87%D0%B8%D1%81%D0%BB%D0%B5%D0%BD%D0%B8%D1%8F%D" ] "_flash" => array:2 [ "old" => [] "new" => [] ] "PHPDEBUGBAR_STACK_DATA" => [] ]