Computer Science / Apache Storm

Real-Time Analytics with Apache Storm

بواسطة: Udacity

Overview

The world is trending in real time! Learn from Twitter to scalably process tweets, or any big data stream, in real-time to drive d3 visualizations using Apache Storm, the "Hadoop of Real Time." Storm is free, open source, and fun to use! Learn from Karthik Ramasamy, Technical Lead of Storm@Twitter, about the distributed, fault-tolerant, and flexible technology used to power Twitter’s real-time data flow pipeline. Twitter open sourced Storm in 2011, and it graduated to a top-level Apache project in September, 2014.

Starting from basic distributed concepts presented during our first Udacity-Twitter Storm Hackathon, link Storm concepts to Storm syntax to scalably drive Word Cloud visualizations with Vagrant, Ubuntu, Maven, Flask, Redis, and d3. Link to the public Twitter gardenhose stream to process live tweets, parse embedded URLs, and calculate Top worldwide hashtags. Extend beyond Storm basics by exploring multi-language capabilities in Python, integrate open source components, and implement real-time streaming joins.

In your final project, follow real-time trending topics by implementing the data pipeline to visualize only tweets that contain Top worldwide hashtags. Extend your project by exploring the Twitter API, or any data source, alongside Hackathon participants as they design their own ideas, receive feedback from Karthik, and open source a final project calculating real-time tweet sentiment and geolocation to drive a U.S. Map.

Syllabus

  • Basic Storm Topologies
    • Link to a real-time d3 Word Cloud Visualization using Redis, Flask, and d3
  • Storm Basics
    • Program Bolts, link Spouts, and connect to the live Twitter API to process real-time tweets,Explore open source components by connecting a Rolling Count Bolt to your topology to visualize Rolling Top Tweeted Words
  • Beyond Storm Basics
    • Explore multi-language capabilities to download and parse real-time Tweeted URLs in Python using Beautiful Soup,Integrate complex open source bolts to calculate Top-N words to visualize real-time Top-N Hashtags,Use stream grouping concepts to easily create streaming join to connect and dynamically process multiple streams
  • Final Project
    • Work on your final project and we cover additional questions and topics brought up by Hackathon participants,Explore Vagrant, VirtualBox, Redis, Flask, and d3 further if you are interested!
  • Final Project: Construct a Storm Topology
    • Design a Storm Topology and new bolt that uses streaming joins to dynamically calculate Top-N Hashtags and display real-time tweets that contain trending Top Hashtags,Post your visualization to the forum and tweet them to your Twitter followers
  • Project Extensions
    • Use additional features of the real-time Twitter sample stream or use any data source to drive your real-time d3 visualization

Taught by

Karthik Ramasamy

Computer Science / Apache Storm
Real-Time Analytics with Apache Storm
الذهاب الي الدورة

Real-Time Analytics with Apache Storm

بواسطة: Udacity

  • Udacity
  • مجانية
  • الإنجليزية
  • متاح شهادة
  • متاح في أي وقت
  • intermediate
  • N/A
MessagesTimelineExceptionsViews14RouteQueries6Models5GateSessionRequest
8.1.2PHP Version1.13sRequest Duration2MBMemory UsageGET ar/الدورات/{slug}Route
    • Booting (739ms)
    • Application (387ms)
    • 1 x Booting (65.5%)
      739.07ms
      1 x Application (34.27%)
      386.64ms
      14 templates were rendered
      • public.courses.show (resources/views/public/courses/show.blade.php)3bladefile
        Params
        0
        course
        1
        links
        2
        config
      • public.courses.partials.breadcrumbs (resources/views/public/courses/partials/breadcrumbs.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.heading (resources/views/public/courses/partials/heading.blade.php)7bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        classes
      • public.courses.partials.details (resources/views/public/courses/partials/details.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.breadcrumbs (resources/views/public/courses/partials/breadcrumbs.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.courses.partials.heading (resources/views/public/courses/partials/heading.blade.php)7bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        classes
      • public.layouts.main (resources/views/public/layouts/main.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.layouts.partials.meta (resources/views/public/layouts/partials/meta.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.layouts.partials.navbar (resources/views/public/layouts/partials/navbar.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.auth.profile.partials.links (resources/views/public/auth/profile/partials/links.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.auth.profile.partials.link (resources/views/public/auth/profile/partials/link.blade.php)8bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
        6
        route
        7
        title
      • public.layouts.partials.flash-session (resources/views/public/layouts/partials/flash-session.blade.php)6bladefile
        Params
        0
        __env
        1
        app
        2
        errors
        3
        course
        4
        links
        5
        config
      uri
      GET ar/الدورات/{slug}
      middleware
      web, localize:ar
      controller
      App\Http\Controllers\CourseController@show
      as
      ar.courses.show
      namespace
      prefix
      /ar
      where
      file
      app/Http/Controllers/CourseController.php:17-35
      6 statements were executed24.48ms
      corspedia
      • select * from `courses` where `slug_ar` = 'real-time-analytics-with-apache-storm' limit 1
        22.56ms/app/Http/Controllers/CourseController.php:20corspedia
        Metadata
        Bindings
        • 0. real-time-analytics-with-apache-storm
        Backtrace
        • 17. /app/Http/Controllers/CourseController.php:20
        • 18. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 19. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 20. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • update `courses` set `visitors` = `visitors` + 1, `courses`.`updated_at` = '2025-06-21 07:43:11' where `id` = 6839
        1.04ms/app/Http/Controllers/CourseController.php:21corspedia
        Metadata
        Bindings
        • 0. 2025-06-21 07:43:11
        • 1. 6839
        Backtrace
        • 17. /app/Http/Controllers/CourseController.php:21
        • 18. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 19. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 20. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select `id`, `name_en`, `name_ar`, `topic_id`, `slug_en`, `slug_ar` from `subjects` where `subjects`.`id` in (131)
        230μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select `id`, `name_en`, `name_ar`, `slug_en`, `slug_ar` from `topics` where `topics`.`id` in (1)
        180μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 25. /app/Http/Controllers/CourseController.php:23
        • 26. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 27. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 28. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 29. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `providers` where `providers`.`id` in (6) and `providers`.`deleted_at` is null
        220μs/app/Http/Controllers/CourseController.php:23corspedia
        Metadata
        Backtrace
        • 20. /app/Http/Controllers/CourseController.php:23
        • 21. /vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54
        • 22. /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:43
        • 23. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:260
        • 24. /vendor/laravel/framework/src/Illuminate/Routing/Route.php:205
      • select * from `html_files` where `html_files`.`id` = 6830 limit 1
        250μs/app/Models/Course.php:84corspedia
        Metadata
        Bindings
        • 0. 6830
        Backtrace
        • 21. /app/Models/Course.php:84
        • 28. view::public.courses.show:29
        • 30. /vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:125
        • 31. /vendor/laravel/framework/src/Illuminate/View/Engines/PhpEngine.php:58
        • 32. /vendor/laravel/framework/src/Illuminate/View/Engines/CompilerEngine.php:72
      App\Models\HtmlFile
      1
      App\Models\Provider
      1
      App\Models\Topic
      1
      App\Models\Subject
      1
      App\Models\Course
      1
        _token
        Slcl5Us6FC5qTF0CRcP2QqlRqHOgezs4cw9XBIoD
        locale
        ar
        _previous
        array:1 [ "url" => "https://www.corspedia.com/ar/%D8%A7%D9%84%D8%AF%D9%88%D8%B1%D8%A7%D8%AA/real-t...
        _flash
        array:2 [ "old" => [] "new" => [] ]
        PHPDEBUGBAR_STACK_DATA
        []
        path_info
        /ar/%D8%A7%D9%84%D8%AF%D9%88%D8%B1%D8%A7%D8%AA/real-time-analytics-with-apache-storm
        status_code
        200
        status_text
        OK
        format
        html
        content_type
        text/html; charset=UTF-8
        request_query
        []
        request_request
        []
        request_headers
          0 of 0   
        array:24 [▼
  "cf-ipcountry" => array:1 [▶
    0 => "US"
  ]
  "cf-connecting-ip" => array:1 [▶
    0 => "216.73.216.35"
  ]
  "cdn-loop" => array:1 [▶
    0 => "cloudflare; loops=1"
  ]
  "x-forwarded-proto" => array:1 [▶
    0 => "https"
  ]
  "x-forwarded-for" => array:1 [▶
    0 => "216.73.216.35"
  ]
  "sec-fetch-site" => array:1 [▶
    0 => "none"
  ]
  "accept" => array:1 [▶
    0 => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
  ]
  "user-agent" => array:1 [▶
    0 => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
  ]
  "upgrade-insecure-requests" => array:1 [▶
    0 => "1"
  ]
  "sec-ch-ua-platform" => array:1 [▶
    0 => ""Windows""
  ]
  "sec-ch-ua-mobile" => array:1 [▶
    0 => "?0"
  ]
  "sec-ch-ua" => array:1 [▶
    0 => ""Chromium";v="130", "HeadlessChrome";v="130", "Not?A_Brand";v="99""
  ]
  "cache-control" => array:1 [▶
    0 => "no-cache"
  ]
  "pragma" => array:1 [▶
    0 => "no-cache"
  ]
  "sec-fetch-dest" => array:1 [▶
    0 => "document"
  ]
  "cf-ray" => array:1 [▶
    0 => "9531df97dcf0f151-ORD"
  ]
  "accept-encoding" => array:1 [▶
    0 => "gzip, br"
  ]
  "priority" => array:1 [▶
    0 => "u=0, i"
  ]
  "sec-fetch-user" => array:1 [▶
    0 => "?1"
  ]
  "sec-fetch-mode" => array:1 [▶
    0 => "navigate"
  ]
  "cf-visitor" => array:1 [▶
    0 => "{"scheme":"https"}"
  ]
  "host" => array:1 [▶
    0 => "www.corspedia.com"
  ]
  "content-length" => array:1 [▶
    0 => ""
  ]
  "content-type" => array:1 [▶
    0 => ""
  ]
]
        request_server
          0 of 0   
        array:50 [▼
  "USER" => "www-data"
  "HOME" => "/var/www"
  "HTTP_CF_IPCOUNTRY" => "US"
  "HTTP_CF_CONNECTING_IP" => "216.73.216.35"
  "HTTP_CDN_LOOP" => "cloudflare; loops=1"
  "HTTP_X_FORWARDED_PROTO" => "https"
  "HTTP_X_FORWARDED_FOR" => "216.73.216.35"
  "HTTP_SEC_FETCH_SITE" => "none"
  "HTTP_ACCEPT" => "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
  "HTTP_USER_AGENT" => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
  "HTTP_UPGRADE_INSECURE_REQUESTS" => "1"
  "HTTP_SEC_CH_UA_PLATFORM" => ""Windows""
  "HTTP_SEC_CH_UA_MOBILE" => "?0"
  "HTTP_SEC_CH_UA" => ""Chromium";v="130", "HeadlessChrome";v="130", "Not?A_Brand";v="99""
  "HTTP_CACHE_CONTROL" => "no-cache"
  "HTTP_PRAGMA" => "no-cache"
  "HTTP_SEC_FETCH_DEST" => "document"
  "HTTP_CF_RAY" => "9531df97dcf0f151-ORD"
  "HTTP_ACCEPT_ENCODING" => "gzip, br"
  "HTTP_PRIORITY" => "u=0, i"
  "HTTP_SEC_FETCH_USER" => "?1"
  "HTTP_SEC_FETCH_MODE" => "navigate"
  "HTTP_CF_VISITOR" => "{"scheme":"https"}"
  "HTTP_HOST" => "www.corspedia.com"
  "REDIRECT_STATUS" => "200"
  "SERVER_NAME" => "corspedia.com"
  "SERVER_PORT" => "443"
  "SERVER_ADDR" => "141.95.147.152"
  "REMOTE_USER" => ""
  "REMOTE_PORT" => "14420"
  "REMOTE_ADDR" => "172.70.131.174"
  "SERVER_SOFTWARE" => "nginx/1.18.0"
  "GATEWAY_INTERFACE" => "CGI/1.1"
  "HTTPS" => "on"
  "REQUEST_SCHEME" => "https"
  "SERVER_PROTOCOL" => "HTTP/2.0"
  "DOCUMENT_ROOT" => "/var/www/corspedia/public"
  "DOCUMENT_URI" => "/index.php"
  "REQUEST_URI" => "/ar/%D8%A7%D9%84%D8%AF%D9%88%D8%B1%D8%A7%D8%AA/real-time-analytics-with-apache-storm"
  "SCRIPT_NAME" => "/index.php"
  "CONTENT_LENGTH" => ""
  "CONTENT_TYPE" => ""
  "REQUEST_METHOD" => "GET"
  "QUERY_STRING" => ""
  "SCRIPT_FILENAME" => "/var/www/corspedia/public/index.php"
  "PATH_INFO" => ""
  "FCGI_ROLE" => "RESPONDER"
  "PHP_SELF" => "/index.php"
  "REQUEST_TIME_FLOAT" => 1750491790.3255
  "REQUEST_TIME" => 1750491790
]
        request_cookies
        []
        response_headers
          0 of 0   
        array:5 [▼
  "content-type" => array:1 [▶
    0 => "text/html; charset=UTF-8"
  ]
  "cache-control" => array:1 [▶
    0 => "no-cache, private"
  ]
  "date" => array:1 [▶
    0 => "Sat, 21 Jun 2025 07:43:11 GMT"
  ]
  "set-cookie" => array:2 [▶
    0 => "XSRF-TOKEN=eyJpdiI6ImR5YURHdzJmMUp4Q05WNUpMTHdCTEE9PSIsInZhbHVlIjoiN0tPRW9IT2ROR2dvT0o1ZDI5c2hBRjAyNWdxYW50TFZCZkcxRkVzbnNHbmpXelJtNVJSUUZFcDVCMXdMSGttRHBCbUdEemJTci9uY2E3dlFUQnp1ak5xWFpsTGl1NytZYzhZb2puSlJiVzQ5dENRM0paWWdRd2V4SlJVRjdtcVYiLCJtYWMiOiIyMmYxZmM4ZGNlNjIxNTU0YTlmYmY0OGNhZjA2YzY5ODYzMDU0ODY3Y2U3NGQ1MzI0YTNiZjlkNTJlYWY0YzEzIiwidGFnIjoiIn0%3D; expires=Sat, 21 Jun 2025 09:43:11 GMT; Max-Age=7200; path=/; samesite=lax ◀XSRF-TOKEN=eyJpdiI6ImR5YURHdzJmMUp4Q05WNUpMTHdCTEE9PSIsInZhbHVlIjoiN0tPRW9IT2ROR2dvT0o1ZDI5c2hBRjAyNWdxYW50TFZCZkcxRkVzbnNHbmpXelJtNVJSUUZFcDVCMXdMSGttRHBCbUdEe ▶"
    1 => "laravel_session=eyJpdiI6IlRUTmZINnVHRTJHQkdFNkZBUHE5bEE9PSIsInZhbHVlIjoiOGN5enE4eEFqcjgreXpTOG84TFQvakJiak9LKzlXeS9XR3VmQWp0UHlhTzEzNWRMaFQzRDFLTW85TTl6aTJxc0JhR0ZXclNqMWdGT3VMc01WMTIzU3dDVEw5S3lWZkVRUW4zUXh0elZEU1FsVU80N0xocXcvRVVhdndJWUtaWjAiLCJtYWMiOiIzYWM2Y2IwODI4OTFlYzc5ZWI0ZmM3YWI5ODkzYzVhNzI1Nzk0NWI2ZDg3ZGFhZDYxZGZjNjgzMmFkN2JmNmMwIiwidGFnIjoiIn0%3D; expires=Sat, 21 Jun 2025 09:43:11 GMT; Max-Age=7200; path=/; httponly; samesite=lax ◀laravel_session=eyJpdiI6IlRUTmZINnVHRTJHQkdFNkZBUHE5bEE9PSIsInZhbHVlIjoiOGN5enE4eEFqcjgreXpTOG84TFQvakJiak9LKzlXeS9XR3VmQWp0UHlhTzEzNWRMaFQzRDFLTW85TTl6aTJxc0Jh ▶"
  ]
  "Set-Cookie" => array:2 [▶
    0 => "XSRF-TOKEN=eyJpdiI6ImR5YURHdzJmMUp4Q05WNUpMTHdCTEE9PSIsInZhbHVlIjoiN0tPRW9IT2ROR2dvT0o1ZDI5c2hBRjAyNWdxYW50TFZCZkcxRkVzbnNHbmpXelJtNVJSUUZFcDVCMXdMSGttRHBCbUdEemJTci9uY2E3dlFUQnp1ak5xWFpsTGl1NytZYzhZb2puSlJiVzQ5dENRM0paWWdRd2V4SlJVRjdtcVYiLCJtYWMiOiIyMmYxZmM4ZGNlNjIxNTU0YTlmYmY0OGNhZjA2YzY5ODYzMDU0ODY3Y2U3NGQ1MzI0YTNiZjlkNTJlYWY0YzEzIiwidGFnIjoiIn0%3D; expires=Sat, 21-Jun-2025 09:43:11 GMT; path=/ ◀XSRF-TOKEN=eyJpdiI6ImR5YURHdzJmMUp4Q05WNUpMTHdCTEE9PSIsInZhbHVlIjoiN0tPRW9IT2ROR2dvT0o1ZDI5c2hBRjAyNWdxYW50TFZCZkcxRkVzbnNHbmpXelJtNVJSUUZFcDVCMXdMSGttRHBCbUdEe ▶"
    1 => "laravel_session=eyJpdiI6IlRUTmZINnVHRTJHQkdFNkZBUHE5bEE9PSIsInZhbHVlIjoiOGN5enE4eEFqcjgreXpTOG84TFQvakJiak9LKzlXeS9XR3VmQWp0UHlhTzEzNWRMaFQzRDFLTW85TTl6aTJxc0JhR0ZXclNqMWdGT3VMc01WMTIzU3dDVEw5S3lWZkVRUW4zUXh0elZEU1FsVU80N0xocXcvRVVhdndJWUtaWjAiLCJtYWMiOiIzYWM2Y2IwODI4OTFlYzc5ZWI0ZmM3YWI5ODkzYzVhNzI1Nzk0NWI2ZDg3ZGFhZDYxZGZjNjgzMmFkN2JmNmMwIiwidGFnIjoiIn0%3D; expires=Sat, 21-Jun-2025 09:43:11 GMT; path=/; httponly ◀laravel_session=eyJpdiI6IlRUTmZINnVHRTJHQkdFNkZBUHE5bEE9PSIsInZhbHVlIjoiOGN5enE4eEFqcjgreXpTOG84TFQvakJiak9LKzlXeS9XR3VmQWp0UHlhTzEzNWRMaFQzRDFLTW85TTl6aTJxc0Jh ▶"
  ]
]
        session_attributes
          0 of 0   
        array:5 [▼
  "_token" => "Slcl5Us6FC5qTF0CRcP2QqlRqHOgezs4cw9XBIoD"
  "locale" => "ar"
  "_previous" => array:1 [▶
    "url" => "https://www.corspedia.com/ar/%D8%A7%D9%84%D8%AF%D9%88%D8%B1%D8%A7%D8%AA/real-time-analytics-with-apache-storm"
  ]
  "_flash" => array:2 [▶
    "old" => []
    "new" => []
  ]
  "PHPDEBUGBAR_STACK_DATA" => []
]