Github: https://github.com/COSMOS-UALR/YoutubeDownload

Author: Joseph Kready

Service:

cosmos-ytdl-1

cosmosadmin

Theualrcosmosadminaccount1

SHH only, Ip can range between144.167.34.58144.167.34.59144.167.34.60144.167.34.61144.167.34.62

Purpose

This service will download youtube videos from COSMOS-DB and separate them into video & audio files. These files will later be used to process movie barcodes and video transcripts

Design

IT services has given us a block of IP addresses (shown above) to use for this project. This is important because these IP addresses are public, so we can switch between them when we get timed out for sending too many requests (404 responses). 

Currently the system is only designed for 1 youtube downloader to run at a time. Future versions could have multiple running, you just need a way to divy up the videos that need to be downloaded. 

Each downloader script is ran in a VM, so that we can set it’s IP address. The files are saved to a different server, to optimize the storage of so many videos. 

Setup

1. Clone the repo
2. Install the requirements
3. Install ffmpeg
4. Mount the drive (see below)
5. Schedule to run using cron tabs

Mounting

Currently, we are storing the videos on COSMOS-2 (144.167.34.243). Below are some instructions on mounting the drive. 

First you need to install cifs utils

sudo mount -t cifs //144.167.34.243/YtDownload /home/cosmosadmin/YoutubeDownload/YtDownload -o user=ytdl,password=Theualrcosmosadminaccount1,file_mode=0777,dir_mode=0777

Starting the script

1. Login to the VM using ssh
2. `cd YoutubeDownload`
3. `conda`
4. `python start.py | tee /home/cosmosadmin/YoutubeDownload/Logs/`date +%Y-%m-%d`.log`