antiwork Reddit Data Set

Project Summary

This repository covers the code and data to generate the antiwork dataset:

• posts to r/antiwork
• from 09/01/21 - 12/31/21

Config software

• cp orig_settings.py settings.py
• Fill in with your reddit API particulars
• Install https://github.com/Jabb0/SubredditDownloader

Harvesting bare posts

Need to download basic metadata of posts using PushShift API

• Invoke on the command line to get the ‘headers’ of post that match above

  SubredditDownloader % subredditdownloader -db ../anti-work.db -r antiwork -ds pushshift --reddit-client-id FILL --reddit-client-secret FILL --reddit-username elibtronic --start-utc 1630468800 --end-utc 1641013200
  

• Use a SQLite browser to open anti-work.db and save as csv file antiwork.csv

Enrich data with freshest API info

Uses the post_id of harvested heads to get full api data using PRAW and save as Pickle file

• Run Inital AP interations.ipynb with grab PRAW objects and dump them into raw_data directory.

Build Working CSV files

Work done in Master_Builder.ipynb.

• Cell with #CONFIGS has all the values you can set for number of records etc
• Cell with #CONFIGS #PICKLE section are the fields from PRAW that you want in final CSV file

Features

• Pulls high, average, low level posts
• Automatically generates keywords based on frequency of terms found in the text
• Adds VADER scoring to records

Example data generated

Header data set

Example PRAW item fetched

Sample of 1000 records as CSV

Analysis Notebooks

Data is built locally. Once that is done analysis it completed in Google Colab according to the following Notebooks:

• Corpus Prep
• Model Build