Assignment 1: Warmup due 11:30am Thurs Sep 17

This assignment is "ready" and has been released.

Assignment Mechanics

High-level overview: We will be using GitHub for the assignments. You will create a private GitHub repo that you will populate throughout the semester for each assignment, successively. You will share your GitHub repo with a service account used for the course. Using this service account, we will pull your repo for grading after the deadline.

‼️ Do not wait until the last minute to go through these steps, as there is a blocker that requires attention from the course staff.

Please follow these instructions exactly:

You need a GitHub account. You can use whatever GitHub account you want — equally okay to use your personal account or a course-specific account.

Create a private GitHub repo, named cs431-2026f-assignments-USERID. Replace USERID with your actual GitHub userid (note, not your UWaterloo directory id). So, for me, since my GitHub userid is lintool, I'd create the repo cs431-2026f-assignments-lintool.

Make sure your repo is private. Otherwise, everyone in the world can see your assignment solutions. Share your repo with the course service account teachtool. That is, add the user as a collaborator. This will give the course staff access to your repo.

Make sure your repo actually exists. Create a stub README.md at the root of your repo to make sure (content doesn't matter).

As a one-time action, we will need to explicitly link your GitHub account to your UWaterloo directory id. That is, we need to know that it is you submitting the assignment. We just need to verify this association for A1.

On Piazza, send the instructors a private message, as follows:

Subject: UWATERLOO_USERID: GitHub account GITHUB_USERID
I am YOUR NAME UWATERLOO_USERID and I certify that my GitHub userid is GITHUB_USERID.

Bind the variables as appropriate. So, for example:

Subject: jimmylin: GitHub account lintool
I am Jimmy Lin jimmylin and I certify that my GitHub userid is lintool.

With this message, you are explicitly confirming your identity and the association between you and your GitHub account. Wait for confirmation from the course staff. We need to make sure we can access your repo.

In your repo, A1 files will go in assignment1/, A2 in assignment2/, etc. You will be using the same repo throughout the semester. Details of what files to submit will be provided in each assignment.

To confirm that everything is in order, check that you can make changes to your own repo. (For example, clone locally, make a small edit, and then push back to origin.)

If you've gotten here, you're done with the "mechanics"... Congrats! 🎉

Let's Actually Get Started

The notebook that contains the actual assignment is here.

Before you start the assignment, make sure you've already installed the software needed for the course: instructions are provided here. You'll be ready to start the assignment once you are able to launch JupyterLab and view the assignment notebook in a browser.

Assignment Submission Details

Here's what you'll turn in: In your assignment repo, you should have a sub-directory called assignment1/. In that directory, you should have the following files:

• assignment1.ipynb: A copy of the assignment notebook, but with code/answers/etc. filled in.
• rdd_top10_stopwords_retained.csv: CSV of top-10 most frequently occurring words, with stopwords retained, using RDDs.
• rdd_top10_stopwords_removed.csv: CSV of top-10 most frequently occurring words, with stopwords removed, using RDDs.
• df_top10_stopwords_retained.csv: CSV of top-10 most frequently occurring words, with stopwords retained, using DataFrames.
• df_top10_stopwords_removed.csv: CSV of top-10 most frequently occurring words, with stopwords removed, using DataFrames.
• sql_top10_stopwords_retained.csv: CSV of top-10 most frequently occurring words, with stopwords retained, using SQL.
• ingest_reviews.py: The data ingestion script.
• README.md: A Markdown file you can use to explain anything that wasn't explicitly requested. Even if you have nothing to say, create an empty stub.

Grading Scheme

• Word count code using RDDs: 6 points
• Word count code using DataFrames: 6 points
• Word count code using SQL: 8 points
• Question in acell_13_3hq81 about tokens: 3 points
• Question in acell_14_3ng52 about RDDs vs. DataFrames: 3 points
• Question in acell_15_3er9s about RDDs vs. SQL: 3 points
• Question in acell_16_1vb6x about the two bugs: 1 point
• Following instructions: 2 points

What does "following instructions" mean? These are "free points" if you follow the instructions provided in this assignment. These points are to handle the scenario where all your answers are correct, but you did not follow the instructions and that caused us to go out of our way to fix your submission so that it conforms to the instructions. (For example, you removed the ids that we used for tracking, which would make it much more difficult to grade.) In these and other related cases, we will dock points from this category.

Total: 32 points