Adwatch

AdWatch is a complete workflow for automatic audio and video content detection, including advertisements and promos.

Stack (OS & PL & FW):

Operating System:

Linux

Programming Language:

Python Framework:

Audfprint, ffmpeg, moviepy

Dataset:

Image dataset of advertisements and news domain .wav files of advertisements for database generation Database containing audio fingerprints of advertisements in a pickle file.

Workflow (Processing):

Video file (.mp4) to audio file (.wav) conversion with standard sample and bitrate Detecting ads by matching their fingerprints from the input file Additional frames detection using CNN Removal of ads from the original input (.mp4) file Output is .mp4 with removed advertisements.

End To End (Development & Integration in a System):

Step1 (Dataset Generation):

For Transfer Learning (VGG-16): .mp4 files of news domain and advertisements with 50:50 or 60:40 time ratio

Frames extraction of both classes

Data annotation

Pre-processing Images according to VGG-16 model

For Audfprint

.mp4 files of advertisements

Conversion of .mp4 file into .wav files with the standard sample rate of 44.1 kHZ and bitrate of 1411k.

Step2 (Training):

Transfer Learning (VGG-16 Model):

Fine-tuned VGG-16 according to our problem and number of classes

Freezed all the layers except the last four layers, which were re-trained for the feature extraction process.

Last output layer neurons were set to 2, as our problem is binary classification.

After training, the best model was saved.

Audfprint:

Generated database of audio fingerprints using following command: ‘python audfprint.py new –dbase database_name.pklz path_of_audio_file/*.mp3 or .wav’

Step3 (Fusion of Audfprint and VGG-16):

Input .mp4 file containing news and ads

Conversion of .mp4 file to .wav file.

Matching maximum ads existing in the file using Audfprint

Updating the time range given by Audfprit through CNN model.

CNN is used to predict the ads and news frames prior to starting time and after the ending time of detected ads.

Removal of ads using starting and ending time.

Output is .mp4 file with all the ads removed.

Step4 (Testing):

Evaluation of Adwatch by feeding in a video stream of news with ads.

Deployment (Server / API):

Output (Screenshots):

Applications (General Real World Use):

Audio and video content detection

Ads verification

Competitive ads analysis

Ads Affidavits reports

Use Case (Our Specific):

To efficiently monitor all media news campaigns i.e., news bulletins, talk shows, press conferences, headlines etc., by removing news commercials that come in the way.