https://maximafoundation.org/ https://gsconsultants.lk/bandarqq/ https://gsconsultants.lk/dominoqq/ slot kamboja slot pulsa slot maxwin pkv games https://revistacipa.com.br/judi-bola-euro-2024/ pkv games https://ashleyadillon.com/ slot77 judi bola slot telkomsel bandarqq https://acapulco.gob.mx/mailer/ https://acapulco.gob.mx/tys/docs/ https://acapulco.gob.mx/pcivil/css/ https://acapulco.gob.mx/pcivil/uac/ https://acapulco.gob.mx/pcivil/recomendaciones/ pkv games dominoqq bonus new member slot depo 5k pkv games slot thailand slot garansi kekalahan slot dana situs toto slot thailand slot gacor mpo play slot bri slot bca slot dana slot mahjong ways slot77 slot77 slot123 pkv games pkv games pkv games pkv games pkv games bandarqq dominoqq dominoqq dominoqq dominoqq MPO mpo play mpo play bandarqq bandarqq pkv games pkv games pkv games jam hoki main slot jam hoki main slot bandarqq bandarqq pkv games bandarqq bandarqq bandarqq bandarqq dominoqq dominoqq dominoqq pkv games pkv games pkv games pokerqq mpo slot indonesia dominoqq bandarqq pkv games bandarqq pkv games pkv games bandarqq dominoqq pkv games pkv games
https://teradata.co.id/dana/ slot thailand https://bpsk.kuningankab.go.id/indosat/ https://aal.ac.id/assets/filemanager/aal/ https://aal.ac.id/assets/filemanager/gacor/ https://aal.ac.id/assets/filemanager/slot-depo-10k/ https://pule-pule.trenggalekkab.go.id/slot-depo-10k/ https://sraten.tuntang.semarangkab.go.id/wp-content/uploads/2023/joker123/ https://sraten.tuntang.semarangkab.go.id/wp-content/uploads/2023/joker123/ https://sraten.tuntang.semarangkab.go.id/wp-content/uploads/2023/sbobet-sbobet88-judi-bola-mix-parlay/
Skip to content

Help Center

Categories
< All Topics
Print

YouTube Automated Collection Process

This document is intended to help you setup the YouTube data collection tool on your local or virtual machine. The YouTube data collection tool utilizes YouTubes data API to get comments, videos and channel information. It is built with python and multi-threaded to increase throughput. An API key is used to authenticate each request to the API.

Before we continue, ensure you have the following setup on your system:

  • IDE e.g., VS Code
  • Python
  • Anaconda
  • Git
  • MySQL Workbench (optional)

Features

  • Get videos based on search keyword
  • Get channels of videos based on search keywords
  • Calculate daily engagement metrics starting from the initial crawl date
  • Get related videos
  • Crawl video and channel information by a list of given IDs

Setup

Clone the repository into your desired folder

  • git clone https://github.com/COSMOS-UALR/YouTubeDataCollection.git
  • cd into the repository
  • cd YouTubeDataCollection

Create virtual environment from environment.yml file and activate the virtual environment

  • conda env create -f environment.yml
  • conda activate daily_crawler

Adding a task/crawl job

A task/crawl job is a project that contains run information required by the crawler and is mapped to actual research projects being conducted. This information is stored under the `task_test` table in the `crawler_task` schema of the COSMOS-DB. Below is the list of parameters used to define the crawl job.

  • task_id: unique id of the project
  • task_name: The name of the project
  • channels: List of Channel IDs to crawl
  • videos: List of Video IDs to crawl
  • keywords: List of YouTube formatted search keywords to get videos relating to the keywords
  • videos_daily: List of videos to get daily engagement and production metrics e.g No. of likes, views
  • get_related_videos: Boolean to get related videos from crawled videos
  • get_comments: Boolean to get comments on videos
  • channel_by_keyword: Boolean to get channel information for videos that match search keywords
  • crawled_from: only get contents published after the specified date

To add a task to the DB open create_task.py and fill in the above parameters -> then run:

                                `python create_task.py`

Confirm that the task was successfully added to the database.

Checking Task Status

The crawler uses the column crawled_time on the crawler_task.task_test table on the COSMOS-DB to keep track of the last time a project is crawled. To check the progress of a crawl job, check the Log_<date>.txt file in the Logs folder to get the progress report on the crawl_job for the specified date.

Adding API Keys:

To add an API Key to the crawler, navigate to the API_Keys/API_Master_keys.txt file and add each key as a new line in the file.

Processing Crawl Jobs:

Execute `run_crawler.py` to process all the tasks added to the `crawler_task` table. The data collection tool runs in an asynchronous fashion. Depending on the parameters specified for a task when it was created, it would get the comments on a video, get new videos by search keywords, get all the videos on a channel, get metadata for specified videos, get channel information, and get channel and videos daily engagement stats.

If a task is being executed for the first time a new schema is created for that task/job in the COSMOS-DB using the task name as the schema name and creating the following tables:

The schema definition used to create each table can be found under the `Sql_template` folder.

Output

The tables on the schema representing a task are populated with the crawled data.

Challenges

There are two common problems that you could face with the crawler.

  • Insufficient API keys to handle all the crawl jobs, thereby causing some tasks not to be started because the API keys have been exhausted.
  • Irrelevant data from YouTube search queries, because the keywords passed, are ambiguous/not precise.

Other Resources:

  • YouTube Data API V3 can be found here
  • Blogpost on creating YouTube API Keys here
Table of Contents

© 2024 Collaboratorium for Social Media and Online Behavioral Studies

pkv games pkv games bandarqq dominoqq pkv games bandarqq dominoqq pkv games bandarqq dominoqq pkv games bandarqq dominoqq pkv games bandarqq dominoqq bandarqq pkv games pkv games bandarqq dominoqq slot depo 5k pkv games pkv games bandarqq dominoqq slot depo 5k pkv games pkv games bandarqq dominoqq slot depo 5k pkv games bandarqq pkv games bandarqq dominoqq pkv games bandarqq dominoqq pkv games bandarqq dominoqq pkv games bandarqq pkv games bandarqq dominoqq pkv games dominoqq pkv games bandarqq dominoqq pkv games bandarqq pkv games bandarqq dominoqq pkv games bandarqq dominoqq bonus new member depo 50 bonus 50 slot telkomsel dominoqq bandarqq slot bca dominoqq slot pulsa slot88 bonus new member depo 25 bonus 25 slot garansi kekalahan slot thailand mpo slot depo 10k slot indosat slot zeus pkv pkv games pkv bandarqq dominoqq pkv games bandarqq dominoqq slot depo 5k mpo slot depo 5k demo slot mahjong kakek merah slot bandarqq pkv dominoqq slot depo 5k slot depo 5k depo 25 bonus 25 depo 50 bonus 50 slot77 bonus new member slot garansi kekalahan pkv games slot dana slot dana slot depo 5k slot garansi kekalahan slot gacor malam ini slot ovo slot77 bonus new member depo 25 bonus 25 bonus new member bandarqq dominoqq slot77 slot77 bandarqq pkv slot gacor bandarqq pkv games bandarqq dominoqq slot gacor slot thailand depo 25 bonus 25 slot77 slot depo 5k slot depo 10k slot depo 25k slot depo 5k slot depo 5k slot depo 10k slot depo 25k slot depo 5k slot depo 5k slot depo 10k slot depo 25k slot depo 5k pkv games bandarqq dominoqq slot depo 5k pkv games dominoqq slot depo 5k slot depo 5k slot gacor thailand pkv games bandarqq dominoqq slot depo 5k slot gacor thailand slot gacor thailand slot depo 5k slot77 pkv games bandarqq dominoqq slot depo 5k slot depo 5k bandarqq pkv games dominoqq bandarqq pkv games dominoqq slot depo 5k dominoqq pkv games slot77 bandarqq dominoqq pkv games depo 25 bonus 25 dominoqq bandarqq pkv games rtp slot pkv games bandarqq dominoqq bocoran admin jarwo pkv games bandarqq dominoqq pkv games bandarqq dominoqq Slot x500 pkv games bandarqq dominoqq Slot77 pkv games bandarqq dominoqq pkv games bandarqq pkv games slot gacor slot online slot88 slot demo slot terbaru slot deposit 5000 rtp slot slot pragmatic slot bonanza slot kakek slot malaysia rtp live slot hoki link daftar slot slot x500 slot77 slot777 mpo slot pkv games bandarqq dominoqq pokerqq qq online slot gacor triofus bocoran admin jarwo bocoran admin riki depo 50 bonus 50 depo 25 bonus 25 bonus new member slot slot telkomsel slot xl slot dana slot bca main slot slot thailand slot maxwin link slot link gacor judi parlay judi bola sbobet slot pulsa slot gampang menang slot demo pragmatic slot deposit pulsa slot 5rb slot 10rb slot 25rb slot 50rb slot bonus new member slot e-wallet situs slot slot indonesia slot server luar akun pro slot server thailand slot bni slot bri slot mandiri slot resmi slot linkaja slot server international pola slot slot mahjong slot kamboja slot filipina slot zeus bo slot nexus slot slot mania slot deposit bank slot new bank slot qris Slot Princes slot petir x500 sbobet88 link judi bola bandar judi bola judi bola liga inggris judi bola liga indonesia judi bola liga luar togel sgp idn slot situs live casino togel macau slot gacor hari ini situs slot gacor link slot gacor deposit pulsa tanpa potongan slot casino situs terpercaya situs slot slot terpercaya slot gacor gampang menang slot gacor 2023 slot deposit pulsa tanpa potongan slot gacor malam ini situs gacor hari ini situs slot terpercaya slot pulsa tanpa potongan situs slot gacor hari ini rtp slot gacor slot luar negeri rtp slot gacor hari ini situs slot terbaik slot paling gacor slot tergacor slot gacor slot online slot88 slot terbaru slot deposit 5000 rtp slot slot pragmatic slot bonanza slot kakek slot malaysia rtp live slot hoki link daftar slot slot x500 slot77 slot777 mpo slot dominoqq pokerqq qq online slot gacor triofus bocoran admin jarwo bocoran admin riki depo 50 bonus 50 depo 25 bonus 25 bonus new member slot slot telkomsel slot xl slot dana slot bca main slot slot thailand slot maxwin link slot link gacor judi parlay judi bola sbobet slot pulsa slot gampang menang slot demo pragmatic slot deposit pulsa slot 5rb slot 10rb slot 25rb slot 50rb slot bonus new member slot e wallet situs slot slot indonesia slot server luar akun pro slot server thailand slot bni slot bri slot mandiri slot resmi slot linkaja slot server international pola slot slot mahjong slot kamboja slot filipina slot zeus bo slot nexus slot slot mania slot deposit bank slot new bank slot qris Slot Princes slot petir x500 sbobet88 link judi bola bandar judi bola judi bola liga inggris judi bola liga indonesia judi bola liga luar togel sgp idn slot situs live casino togel macau slot gacor hari ini situs slot gacor link slot gacor deposit pulsa tanpa potongan slot casino situs terpercaya situs slot slot terpercaya slot gacor gampang menang slot gacor 2023 slot deposit pulsa tanpa potongan slot gacor malam ini situs gacor hari ini situs slot terpercaya slot pulsa tanpa potongan situs slot gacor hari ini rtp slot gacor pkv games slot gacor bandarqq pkv games bandarqq dominoqq slot gacor slot online slot88 slot demo slot terbaru slot deposit 5000 rtp slot slot pragmatic slot bonanza slot kakek slot malaysia rtp live slot hoki link daftar slot slot x500 slot77 slot777 mpo slot pkv games bandarqq dominoqq pokerqq qq online slot gacor triofus bocoran admin jarwo bocoran admin riki depo 50 bonus 50 depo 25 bonus 25 bonus new member slot slot telkomsel slot xl slot dana slot bca main slot slot thailand slot maxwin link slot link gacor judi parlay judi bola situs judi bola judi bola terpercaya situs judi bola resmi judi bola parlay situs judi bola resmi di indonesia situs judi bola online agen judi bola terpercaya situs judi bola terbesar di indonesia pasaran judi bola daftar judi bola cara main judi bola pemula web judi bola judi bola dunia prediksi judi bola malam ini daftar judi bola 88 akun judi bola website judi bola situs judi bola resmi dan terpercaya prediksi judi bola pasaran judi bola liga 1 indonesia bandar judi bola terbesar di dunia over under judi bola judi bola indonesia judi bola sbobet slot judi bola judi bola terbaik aplikasi judi bola situs judi bola terbesar di asia daftar situs judi bola resmi judi bola slot situs judi bola resmi asia bandar judi bola situs judi bola terpercaya agen judi bola judi bola online judi bola piala dunia judi bola resmi parlay judi bola link judi bola situs judi bola terbesar sbobet slot pulsa slot gampang menang slot demo pragmatic slot deposit pulsa slot 5rb slot 10rb slot 25rb slot 50rb slot bonus new member slot e-wallet situs slot slot indonesia slot server luar akun pro slot server thailand slot bni slot bri slot mandiri slot resmi slot linkaja slot server international pola slot slot mahjong slot kamboja slot filipina slot zeus bo slot nexus slot slot mania slot deposit bank slot new bank slot qris Slot Princes slot petir x500 sbobet88 link judi bola bandar judi bola judi bola liga inggris judi bola liga indonesia judi bola liga luar togel sgp idn slot situs live casino togel macau slot gacor hari ini situs slot gacor slot demo gacor judi slot gacor triofus link slot gacor slot gacor maxwin gacor x500 situs gacor deposit pulsa tanpa potongan slot casino situs terpercaya situs slot slot terpercaya slot gacor gampang menang slot gacor 2023 slot deposit pulsa tanpapotongan slot gacor malam ini situs gacor hari ini situs slot terpercaya slot pulsa tanpa potongan situs slot gacor hari ini rtp slot gacor slot luar negeri rtp slot gacor hari ini situs slot terbaik slot paling gacor slot tergacor situs gacor malam ini gate of olympus slotdepo 5k rtp gacor rtp gacor hari ini pola gacor olympus deposit pulsa slot server thailand super gacor bo slot gacor link gacor slot info slot gacor gacor slot pola gacor hari ini daftar slot slot online gacor link slot gacor thailand jam gacor slot slot gacor hari ini pragmatic akun slot gacor situs judi slot gacor mpo gacor bocoran slot gacor daftar slot gacor pola slot gacor hari ini slot gacor minimal deposit 5rb pola slot gacor link gacor hari ini info slot gacor hari ini demo slot gacor bocoran slot gacor hari ini slot game situs slot gacor terpercaya slot depo 10k slot gacor terpercaya link slot gacor hari ini slot gacor terbaru daftar slot online slot online terpercaya agen slot thailand akun gacor thailand server gacor thailand agen slot server thailand link slot situs thailand akun slot luar thailand super thailand slot akun slot pro thailand slot akun thailand gacor th