Script to automatically tweet new blog posts based in a website RSS feed

Overview

Based in the RSS feed generated by websites, feed a Twitter account with daily new blog posts with Bash and Python.

This method is specially useful for websites that has been built with static site generators, like Jekyll for example, where they automatically publish a RSS feed with its new posts in each build.

Based in this information, you can set up a script in another server that will check the above feed, and if it detects new posts it will publish a link to it directly in Twitter.

Process

Set up a virtual environment

If you already have a virtual environment installed, you can skip this step. In Ubuntu you can install the package python-virtualenv.

$ apt install python-virtualenv

Activate the virtual environment to install the required packages

$ virtualenv -p python3.6 ~/.virtualenvs/twitter_bot
Running virtualenv with interpreter /usr/bin/python3.6
New python executable in /home/user/.virtualenvs/twitter_bot/bin/python3.6
Also creating executable in /home/user/.virtualenvs/twitter_bot/bin/python

$ source ~/.virtualenvs/twitter_bot/bin/activate
(twitter_bot)$

Twitter credentials

Log into Twitter Apps and select “Create New App”.

Create the Application with the following content:

Name

Website RSS Twitter Feeder

Description

Automatic new blog posts publisher

Website

Example.com

Callback URL

And then proceed to “Create your Twitter Application” button.

Now go to “Keys and Access Tokens” tab, and make sure your app has rights to publish tweets, the Access Level Read and write option should be enabled.

Then go to “Token Actions” and “Create my access token”.

You will need the following items from this page:

• Consumer Key (API Key)
• Consumer Secret (API Secret)
• Access Token
• Access Token Secret

Set up application

We will use the publishfeed script to publish tweets, so following its instructions, we clone the repo:

(twitter_bot)$ git clone https://github.com/marcanuy/publishfeed
Cloning into 'publishfeed'...
remote: Counting objects: 41, done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 41 (delta 13), reused 34 (delta 9), pack-reused 0
Unpacking objects: 100% (41/41), done.
(twitter_bot)$ cd publishfeed
(twitter_bot) publishfeed$

Set up feeds

Copy feeds.yml.skel to feeds.yml and edit your feeds information:

(twitter_bot) publishfeed$ cd publishfeed
(twitter_bot) publishfeed$ cp feeds.yml.skel feeds.yml

Install dependencies

Dependencies are handled by pip:

$ make install
pip install -r requirements.txt
Collecting beautifulsoup4==4.6.0 (from -r requirements.txt (line 1))
  Using cached beautifulsoup4-4.6.0-py3-none-any.whl
Collecting certifi==2017.4.17 (from -r requirements.txt (line 2))
  Using cached certifi-2017.4.17-py2.py3-none-any.whl
Collecting chardet==3.0.3 (from -r requirements.txt (line 3))
  Using cached chardet-3.0.3-py2.py3-none-any.whl
Collecting feedparser==5.2.1 (from -r requirements.txt (line 4))
Collecting idna==2.5 (from -r requirements.txt (line 5))
  Using cached idna-2.5-py2.py3-none-any.whl
Collecting munch==2.1.1 (from -r requirements.txt (line 6))
Collecting oauthlib==2.0.2 (from -r requirements.txt (line 7))
Collecting PyYAML==3.12 (from -r requirements.txt (line 8))
Collecting requests==2.17.3 (from -r requirements.txt (line 9))
  Using cached requests-2.17.3-py2.py3-none-any.whl
Collecting requests-oauthlib==0.8.0 (from -r requirements.txt (line 10))
  Using cached requests_oauthlib-0.8.0-py2.py3-none-any.whl
Collecting six==1.10.0 (from -r requirements.txt (line 11))
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting SQLAlchemy==1.1.10 (from -r requirements.txt (line 12))
Collecting tweepy==3.5.0 (from -r requirements.txt (line 13))
  Using cached tweepy-3.5.0-py2.py3-none-any.whl
Collecting urllib3==1.21.1 (from -r requirements.txt (line 14))
  Using cached urllib3-1.21.1-py2.py3-none-any.whl
Installing collected packages: beautifulsoup4, certifi, chardet, feedparser, idna, six, munch, oauthlib, PyYAML, urllib3, requests, requests-oauthlib, SQLAlchemy, tweepy
Successfully installed PyYAML-3.12 SQLAlchemy-1.1.10 beautifulsoup4-4.6.0 certifi-2017.4.17 chardet-3.0.3 feedparser-5.2.1 idna-2.5 munch-2.1.1 oauthlib-2.0.2 requests-2.17.3 requests-oauthlib-0.8.0 six-1.10.0 tweepy-3.5.0 urllib3-1.21.1

Download feeds

After setting up credentials you can execute python main.py –getfeeds And then tweet each post: python main.py –tweet

(twitter_bot) feedr$ python main.py TWITTERHANDLER --getfeeds
(twitter_bot) feedr/src$ python main.py TWITTERHANDLER --tweet

Schedule posts

We set up two cron jobs, one to download new posts in each feed, in this case every hour, and another one that will tweet one of each post when we execute it, so it can be good to set it up with for example, 15 minutes between each run. We enter the crontab editor:

$ crontab -e

And then we add the following line (adjust the path of your installation, in this case I have used /opt/publishfeed)

# download feeds hourly
0 * * * * cd /opt/publishfeed/publishfeed/; flock -n /tmp/twsimpleitp.lock ~/.virtualenvs/publishfeed/bin/python main.py simpleitrocks -g
# publish tweets every 15 minutes
*/15 * * * * cd /opt/publishfeed/publishfeed/; flock -n /tmp/twsimpleitp.lock ~/.virtualenvs/publishfeed/bin/python main.py simpleitrocks -t

We use the flock command to prevent duplicate cron job executions.

Conclusion

Every time the cron job detects new content present in a website feed it will tweet in the account you have selected. This is a great way to have your blog posts automatically tweeted and your content always present in social media.

Reference

• https://en.wikipedia.org/wiki/Web_feed
• https://en.wikipedia.org/wiki/RSS
• https://github.com/housed/feedr

*[RSS]: Really Simple Syndication