Twitter Research Summary
This was month-long project that allowed me to build and deploy a Python/Django/PostgreSQL web application. The application is using out-of-the-box Django Templates and JQuery only for the UI. It is deployed to one Digital Ocean droplet using Nginx, Lets Encrypt, Gunicorn. It is also using a managed PostgreSQL database that resides in a Digital Ocean managed database server.
The app allows users to enter a Twitter search term and date range. It authenticates to Twitter using keys I acquired from Twitter a couple years ago. If the user has requested counts, the counts endpoint is used to return counts for each day in the request period. If the user requested data, it makes a request to either the 30-day or full-archive endpoints depending on the date range entered.
What? I can’t register?
No registration method has been provided. Twitter data has limits and I don’t want people eating up the limit without my knowledge. If someone wants to use it, they need to contact me and I’ll set up a login for them.
All requests return a .csv file.
What went well
I discovered Digital Ocean. I think this is a really good alternative to AWS for a small personal project like this. I’m able to host a Postgresql database for $15/mo. All of the pieces of the web site are on one server. I found a good tutorial that runs through all the steps to Dockerize the app and deploy it (they used AWS) with Nginx and LetsEncrypt. Even though I’ve never really built tests for a Django project before, I used this project as an opportunity to learn how that all works. I love that the tests are there and that I now know how to set them up and that Django is so testable.
With Digital Ocean, you know the upper limits of your costs!
I very much enjoyed the test-driven.io tutorials. They were exactly what I needed. Digital Ocean also has some good information in their tutorials that helped me. In addition, Digital Ocean’s tutorials often come with the ability to ask questions and give feed back. Even though the test-driven.io tutorials didn’t really provide information about what people who used them had trouble with, they were very beneficial to me.
Due to factors that didn’t go so well (see below), I ended up getting what I think is a pretty good domain name for the site—tweetsforresearch.com.
What went less than well
Omg on the AWS and Docker training. I took both of those way too far. Not saying that the courses I chose were not good. There are better ways to learn, though (did I already say I loved the test-driven.io tutorials)?
I stopped working on this tutorial before its end. Not sure what I was thinking (I think I may have taken a two-day grand-kid break), but that cost me some time.
Even though the tutorial talked about limits on LetsEncrypt, I made the mistake of changing from the Staging environment to production too soon, so https didn’t work on my original domain of choice—susanenneking.dev. I was forced to look for another domain because of this.
Still Missing
The end product is missing an up-to-date Javascript library. It’s also missing the final Terraform version of the set-up. I’ll work on those next.
Because I am now, officially, looking for employment, I’m going to start with adding React. I’ll do the Terraform set-up after that.
Then, of course, there’s the whole organize the data to help the researcher part. Melody has given her master’s thesis text and a book about qualitative analysis to help me understand what she did to sort out the useful/non-useful data. That will be the next big enhancement.
That was fun! I’m going to do this type of thing more often–actually, continuously!