Robotics: Perception

Brought by: Coursera

Overview

How can robots perceive the world and their own movements so that they accomplish navigation and manipulation tasks? In this module, we will study how images and videos acquired by cameras mounted on robots are transformed into representations like features and optical flow. Such 2D representations allow us then to extract 3D information about where the camera is and in which direction the robot moves. You will come to understand how grasping objects is facilitated by the computation of 3D posing of objects and navigation can be accomplished by visual odometry and landmark-based localization.

Syllabus

  • Geometry of Image Formation
    • Welcome to Robotics: Perception! We will begin this course with a tutorial on the standard camera models used in computer vision. These models allow us to understand, in a geometric fashion, how light from a scene enters a camera and projects onto a 2D image. By defining these models mathematically, we will be able understand exactly how a point in 3D corresponds to a point in the image and how an image will change as we move a camera in a 3D environment. In the later modules, we will be able to use this information to perform complex perception tasks such as reconstructing 3D scenes from video.
  • Projective Transformations
    • Now that we have a good camera model, we will explore the geometry of perspective projections in depth. We will find that this projection is the cause of the main challenge in perception, as we lose a dimension that we can no longer directly observe. In this module, we will learn about several properties of projective transformations in depth, such as vanishing points, which allow us to infer complex information beyond our basic camera model.
  • Pose Estimation
    • In this module we will be learning about feature extraction and pose estimation from two images. We will learn how to find the most salient parts of an image and track them across multiple frames (i.e. in a video sequence). We will then learn how to use features to find the position of the camera with respect to another reference frame on a plane using Homographies. We will also learn about how to make these techniques more robust, using least squares to hand noisy feature points or RANSAC to remove completely erroneous feature points.
  • Multi-View Geometry
    • Now we will use what we learned from two view geometry and extend it to sequences of images, such as a video. We will explain the fundamental geometric constraints between point features in images, the Epipolar constraint, and learn how to use it to extract the relative poses between multiple frames. We will finish by combining all this information together for the application of Structure from Motion, where we will compute the trajectory of a camera and a map throughout many frames and refine our estimates using Bundle adjustment.

Taught by

Kostas Daniilidis and Jianbo Shi

Robotics: Perception
Go to course

Robotics: Perception

Brought by: Coursera

  • Coursera
  • Free
  • English
  • Certificate Available
  • Available at any time
  • intermediate
  • French, Portuguese, Russian, English, Spanish, Chinese, Kazakh, German, Hindi, Swedish, Korean, Greek, Italian, Arabic, Thai, Ukrainian, Japanese, Indonesian, Polish, Dutch, Turkish
8.1.2PHP Version246msRequest Duration2MBMemory UsageGET en/courses/{slug}Route
    • Booting (149ms)
    • Application (96.51ms)
    • 1 x Booting (60.51%)
      148.80ms
      1 x Application (39.24%)
      96.51ms
      14 templates were rendered
      • public.courses.show (resources/views/public/courses/show.blade.php)3bladefile
        Params
        0
        course
        1
        links
        2
        config
      • public.courses.partials.breadcrumbs (resources/views/public/courses/partials/breadcrumbs.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.heading (resources/views/public/courses/partials/heading.blade.php)7bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        classes
      • public.courses.partials.details (resources/views/public/courses/partials/details.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.breadcrumbs (resources/views/public/courses/partials/breadcrumbs.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.heading (resources/views/public/courses/partials/heading.blade.php)7bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        classes
      • public.layouts.main (resources/views/public/layouts/main.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.layouts.partials.meta (resources/views/public/layouts/partials/meta.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.layouts.partials.navbar (resources/views/public/layouts/partials/navbar.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.auth.profile.partials.links (resources/views/public/auth/profile/partials/links.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.layouts.partials.flash-session (resources/views/public/layouts/partials/flash-session.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      uri
      GET en/courses/{slug}
      middleware
      web, localize:en
      controller
      App\Http\Controllers\CourseController@show
      as
      en.courses.show
      namespace
      prefix
      /en
      where
      file
      app/Http/Controllers/CourseController.php:17-35
      7 statements were executed4.55ms
      • select * from `courses` where `slug_en` = 'robotics:-perception' limit 1
        3.05ms/app/Http/Controllers/CourseController.php:20corspedia
        Metadata
        Bindings
        • 0. robotics:-perception
        Backtrace
        • 17. /app/Http/Controllers/CourseController.php:20
        • 18. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 19. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 20. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • update `courses` set `visitors` = `visitors` + 1, `courses`.`updated_at` = '2025-06-08 22:28:36' where `id` = 65
        530μs/app/Http/Controllers/CourseController.php:21corspedia
        Metadata
        Bindings
        • 0. 2025-06-08 22:28:36
        • 1. 65
        Backtrace
        • 17. /app/Http/Controllers/CourseController.php:21
        • 18. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 19. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 20. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select `id`, `name_en`, `name_ar`, `topic_id`, `slug_en`, `slug_ar` from `subjects` where `subjects`.`id` in (16)
        200μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select `id`, `name_en`, `name_ar`, `slug_en`, `slug_ar` from `topics` where `topics`.`id` in (1)
        140μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 25. /app/Http/Controllers/CourseController.php:23
        • 26. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 27. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 28. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 29. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `institutions` where `institutions`.`id` in (23) and `institutions`.`deleted_at` is null
        190μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `providers` where `providers`.`id` in (2) and `providers`.`deleted_at` is null
        170μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `html_files` where `html_files`.`id` = 65 limit 1
        270μs/app/Models/Course.php:84corspedia
        Metadata
        Bindings
        • 0. 65
        Backtrace
        • 21. /app/Models/Course.php:84
        • 28. view::public.courses.show:29
        • 30. /vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:125
        • 31. /vendor/laravel/framework/src/Illuminate/View/Engines/PhpEngine.php:58
        • 32. /vendor/laravel/framework/src/Illuminate/View/Engines/CompilerEngine.php:72
      App\Models\HtmlFile
      1
      App\Models\Provider
      1
      App\Models\Institution
      1
      App\Models\Topic
      1
      App\Models\Subject
      1
      App\Models\Course
      1
        _token
        1TBcosT0MMY488QYgHZSs1j4fvOFH99C2yUWq35W
        locale
        en
        _previous
        array:1 [ "url" => "https://www.corspedia.com/en/courses/robotics:-perception" ]
        _flash
        array:2 [ "old" => [] "new" => [] ]
        PHPDEBUGBAR_STACK_DATA
        []
        path_info
        /en/courses/robotics:-perception
        status_code
        200
        
        status_text
        OK
        format
        html
        content_type
        text/html; charset=UTF-8
        request_query
        []
        
        request_request
        []
        
        request_headers
        0 of 0
        array:24 [ "cf-ipcountry" => array:1 [ 0 => "US" ] "cf-connecting-ip" => array:1 [ 0 => "216.73.216.123" ] "cdn-loop" => array:1 [ 0 => "cloudflare; loops=1" ] "x-forwarded-proto" => array:1 [ 0 => "https" ] "x-forwarded-for" => array:1 [ 0 => "216.73.216.123" ] "sec-fetch-site" => array:1 [ 0 => "none" ] "accept" => array:1 [ 0 => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" ] "user-agent" => array:1 [ 0 => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" ] "upgrade-insecure-requests" => array:1 [ 0 => "1" ] "sec-ch-ua-platform" => array:1 [ 0 => ""Windows"" ] "sec-ch-ua-mobile" => array:1 [ 0 => "?0" ] "sec-ch-ua" => array:1 [ 0 => ""Chromium";v="130", "HeadlessChrome";v="130", "Not?A_Brand";v="99"" ] "cache-control" => array:1 [ 0 => "no-cache" ] "pragma" => array:1 [ 0 => "no-cache" ] "sec-fetch-dest" => array:1 [ 0 => "document" ] "cf-ray" => array:1 [ 0 => "94cbd2bd2de5f855-ORD" ] "accept-encoding" => array:1 [ 0 => "gzip, br" ] "priority" => array:1 [ 0 => "u=0, i" ] "sec-fetch-user" => array:1 [ 0 => "?1" ] "sec-fetch-mode" => array:1 [ 0 => "navigate" ] "cf-visitor" => array:1 [ 0 => "{"scheme":"https"}" ] "host" => array:1 [ 0 => "www.corspedia.com" ] "content-length" => array:1 [ 0 => "" ] "content-type" => array:1 [ 0 => "" ] ]
        request_server
        0 of 0
        array:50 [ "USER" => "www-data" "HOME" => "/var/www" "HTTP_CF_IPCOUNTRY" => "US" "HTTP_CF_CONNECTING_IP" => "216.73.216.123" "HTTP_CDN_LOOP" => "cloudflare; loops=1" "HTTP_X_FORWARDED_PROTO" => "https" "HTTP_X_FORWARDED_FOR" => "216.73.216.123" "HTTP_SEC_FETCH_SITE" => "none" "HTTP_ACCEPT" => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" "HTTP_USER_AGENT" => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" "HTTP_UPGRADE_INSECURE_REQUESTS" => "1" "HTTP_SEC_CH_UA_PLATFORM" => ""Windows"" "HTTP_SEC_CH_UA_MOBILE" => "?0" "HTTP_SEC_CH_UA" => ""Chromium";v="130", "HeadlessChrome";v="130", "Not?A_Brand";v="99"" "HTTP_CACHE_CONTROL" => "no-cache" "HTTP_PRAGMA" => "no-cache" "HTTP_SEC_FETCH_DEST" => "document" "HTTP_CF_RAY" => "94cbd2bd2de5f855-ORD" "HTTP_ACCEPT_ENCODING" => "gzip, br" "HTTP_PRIORITY" => "u=0, i" "HTTP_SEC_FETCH_USER" => "?1" "HTTP_SEC_FETCH_MODE" => "navigate" "HTTP_CF_VISITOR" => "{"scheme":"https"}" "HTTP_HOST" => "www.corspedia.com" "REDIRECT_STATUS" => "200" "SERVER_NAME" => "corspedia.com" "SERVER_PORT" => "443" "SERVER_ADDR" => "141.95.147.152" "REMOTE_USER" => "" "REMOTE_PORT" => "16044" "REMOTE_ADDR" => "172.70.130.225" "SERVER_SOFTWARE" => "nginx/1.18.0" "GATEWAY_INTERFACE" => "CGI/1.1" "HTTPS" => "on" "REQUEST_SCHEME" => "https" "SERVER_PROTOCOL" => "HTTP/2.0" "DOCUMENT_ROOT" => "/var/www/corspedia/public" "DOCUMENT_URI" => "/index.php" "REQUEST_URI" => "/en/courses/robotics:-perception" "SCRIPT_NAME" => "/index.php" "CONTENT_LENGTH" => "" "CONTENT_TYPE" => "" "REQUEST_METHOD" => "GET" "QUERY_STRING" => "" "SCRIPT_FILENAME" => "/var/www/corspedia/public/index.php" "PATH_INFO" => "" "FCGI_ROLE" => "RESPONDER" "PHP_SELF" => "/index.php" "REQUEST_TIME_FLOAT" => 1749421716.2903 "REQUEST_TIME" => 1749421716 ]
        request_cookies
        []
        
        response_headers
        0 of 0
        array:5 [ "content-type" => array:1 [ 0 => "text/html; charset=UTF-8" ] "cache-control" => array:1 [ 0 => "no-cache, private" ] "date" => array:1 [ 0 => "Sun, 08 Jun 2025 22:28:36 GMT" ] "set-cookie" => array:2 [ 0 => "XSRF-TOKEN=eyJpdiI6IjNYOS9LWmp2Vy80UmtYMUYvaVRpdkE9PSIsInZhbHVlIjoiQVNEUjR0a1VOcUI5RzlrS3ZaNVhLNDl1UFlGbkdoZnlnM0FXM2F4cXNkN0ozeHNLemlUa05RRnZQQTVKRzJSbURYdW0wN2c4RVR6dnZXVzZnc1NyY1U4UVV4bXRXR2c3cVNKSEhjS0w0U0ptMjBWZ3lsdDI5cFhiNjJMbjR6bVciLCJtYWMiOiJlNmU1ZTM2OGJkYWViZmY3NTE1N2I4NTJmYzE0MGU0ZjA4YzNlNTk0MDQ4MTc3ZmUxZjMwZDA5Y2NkOGRjMzk5IiwidGFnIjoiIn0%3D; expires=Mon, 09 Jun 2025 00:28:36 GMT; Max-Age=7200; path=/; samesite=laxXSRF-TOKEN=eyJpdiI6IjNYOS9LWmp2Vy80UmtYMUYvaVRpdkE9PSIsInZhbHVlIjoiQVNEUjR0a1VOcUI5RzlrS3ZaNVhLNDl1UFlGbkdoZnlnM0FXM2F4cXNkN0ozeHNLemlUa05RRnZQQTVKRzJSbURYdW0wN" 1 => "laravel_session=eyJpdiI6IldscFJxWDl3WTJnZUluV1UxUTdhdUE9PSIsInZhbHVlIjoiV1pYa3RxeFJIdzRWa3ZvbXlDdEQzQ1U4VGxoVjZENlNVbk8vSlUwM21yY3lHelVuTHlOd01mSGdXaWYwZXpRWnEzcDRDYk82OExOUnhGZXlVRWlTVXFxM2taMWtlQTdRZVV6OEpCUDd4a2ZsRk5vUm9hMkZXeVd4NHlVaWNNMEgiLCJtYWMiOiJhMDQ4YWE5YTgxZmFiOTE5MDlhYWQzM2M4NDI3NjU3NmMzYmQ1OWFmMGJkMzZiMmRkM2NjZDIzYTc2OWFmN2EwIiwidGFnIjoiIn0%3D; expires=Mon, 09 Jun 2025 00:28:36 GMT; Max-Age=7200; path=/; httponly; samesite=laxlaravel_session=eyJpdiI6IldscFJxWDl3WTJnZUluV1UxUTdhdUE9PSIsInZhbHVlIjoiV1pYa3RxeFJIdzRWa3ZvbXlDdEQzQ1U4VGxoVjZENlNVbk8vSlUwM21yY3lHelVuTHlOd01mSGdXaWYwZXpRWnEz" ] "Set-Cookie" => array:2 [ 0 => "XSRF-TOKEN=eyJpdiI6IjNYOS9LWmp2Vy80UmtYMUYvaVRpdkE9PSIsInZhbHVlIjoiQVNEUjR0a1VOcUI5RzlrS3ZaNVhLNDl1UFlGbkdoZnlnM0FXM2F4cXNkN0ozeHNLemlUa05RRnZQQTVKRzJSbURYdW0wN2c4RVR6dnZXVzZnc1NyY1U4UVV4bXRXR2c3cVNKSEhjS0w0U0ptMjBWZ3lsdDI5cFhiNjJMbjR6bVciLCJtYWMiOiJlNmU1ZTM2OGJkYWViZmY3NTE1N2I4NTJmYzE0MGU0ZjA4YzNlNTk0MDQ4MTc3ZmUxZjMwZDA5Y2NkOGRjMzk5IiwidGFnIjoiIn0%3D; expires=Mon, 09-Jun-2025 00:28:36 GMT; path=/XSRF-TOKEN=eyJpdiI6IjNYOS9LWmp2Vy80UmtYMUYvaVRpdkE9PSIsInZhbHVlIjoiQVNEUjR0a1VOcUI5RzlrS3ZaNVhLNDl1UFlGbkdoZnlnM0FXM2F4cXNkN0ozeHNLemlUa05RRnZQQTVKRzJSbURYdW0wN" 1 => "laravel_session=eyJpdiI6IldscFJxWDl3WTJnZUluV1UxUTdhdUE9PSIsInZhbHVlIjoiV1pYa3RxeFJIdzRWa3ZvbXlDdEQzQ1U4VGxoVjZENlNVbk8vSlUwM21yY3lHelVuTHlOd01mSGdXaWYwZXpRWnEzcDRDYk82OExOUnhGZXlVRWlTVXFxM2taMWtlQTdRZVV6OEpCUDd4a2ZsRk5vUm9hMkZXeVd4NHlVaWNNMEgiLCJtYWMiOiJhMDQ4YWE5YTgxZmFiOTE5MDlhYWQzM2M4NDI3NjU3NmMzYmQ1OWFmMGJkMzZiMmRkM2NjZDIzYTc2OWFmN2EwIiwidGFnIjoiIn0%3D; expires=Mon, 09-Jun-2025 00:28:36 GMT; path=/; httponlylaravel_session=eyJpdiI6IldscFJxWDl3WTJnZUluV1UxUTdhdUE9PSIsInZhbHVlIjoiV1pYa3RxeFJIdzRWa3ZvbXlDdEQzQ1U4VGxoVjZENlNVbk8vSlUwM21yY3lHelVuTHlOd01mSGdXaWYwZXpRWnEz" ] ]
        session_attributes
        0 of 0
        array:5 [ "_token" => "1TBcosT0MMY488QYgHZSs1j4fvOFH99C2yUWq35W" "locale" => "en" "_previous" => array:1 [ "url" => "https://www.corspedia.com/en/courses/robotics:-perception" ] "_flash" => array:2 [ "old" => [] "new" => [] ] "PHPDEBUGBAR_STACK_DATA" => [] ]