Using octomachinery to respond to GitHub events =============================================== In the previous example, we've been interacting with GitHub by making requests to GitHub. And we've been doing that locally on our own machine. In this section we'll use what we know so far and start building an actual bot. We'll create a webserver that responds to GitHub webhook events. Webhook events -------------- When an event is triggered in GitHub, GitHub can notify you about the event by sending you a POST HTTP request along with the payload. Some example ``events`` are: - issues: any time an issue is assigned, unassigned, labeled, unlabeled, opened, edited, closed, reopened, etc. - pull_request: any time a pull request is opened, edited, closed, reopened, review requested, etc. - status: any time there's status update. - checks: any time there's check requested or its status update posted. - installations: any time GitHub App is installed or removed from a repo or new update permissions are accepted by the user. The complete list of events is listed `here `_. Since GitHub needs to send you POST requests for the webhook, it can't send them to your personal laptop. So we need to create a webservice that's open on the Internet. To the cloud! ☁ Create a new Heroku App ----------------------- Login to your account on Heroku. You should land at https://dashboard.heroku.com/apps. Click "New" > `"Create a new app" `_. Type in the app name and click "Create app" button. If you leave it empty, Heroku will assign a name for you. On the app page, right-click on the "Open app" button and click "Copy link address". We'll need this a bit later, in the next step. `Create a new GitHub App`_ -------------------------- 1. Go to `profile Settings`_ > `Developer Settings`_ > `GitHub Apps`_ > `New GitHub App`_ 2. Click `"New GitHub App" `_. 3. Fill in 3 required fields: - Webhook URL (*paste the URL of the Heroku app you copied earlier*, it should look smth like ``https://your-heroku-app-name.herokuapp.com/``) - GitHub App name (this will be turned into an app slug) - Homepage URL (put some URL here, you might use the same as for webhooks and change it later) - **(New!)** Enable the checkbox called **Active** 4. Select the following permissions: - ``Checks``: ``Read & Write`` - ``Issues``: ``Read & Write`` - ``Pull requests``: ``Read & Write`` 5. Go to "Subscribe to events" section and select: - ``Check run`` - ``Check suite`` - ``Issue comment`` - ``Issues`` - ``Label`` - ``Pull request`` 6. Keep ``Only on this account`` radio selected and hit "Create GitHub App" 7. Click "Generate a private key" in the bottom of the app page and download it onto your computer 8. Keep the app page open, you'll need it soon Create a new repository on GitHub --------------------------------- You'll need a GitHub repository. We'll store the bot source code there which is essentially a webservice application. Further in this tutorial, it'll be referred to as ``github-bot``. Later we will also use this same repo to test our bot — we'll install a GitHub App there and it'll be able to interact with this repo. You can create the repository on GitHub, and then clone it to your local machine. Otherwise, ensure that you know how to add a remote and push a pre-existing Git repo to GitHub. .. note:: **Pro tip:** Use dashes (kebab-case) for your repo name like ``github-bot`` or ``pycon-2020-github-app`` so that this name is not a valid Python importable. This'll save you from confusion when you're running commands from a wrong directory (accidentally). Create a github-bot ------------------- Let's get ready to write your own GitHub bot. To start, use your favorite text editor or IDE. Go to the directory where your github-bot is at (the root of the repository you've created earlier). Inside that directory, create a ``requirements.txt`` file. Add ``octomachinery`` to it. ``requirements.txt``: .. literalinclude:: resources/github-bot/requirements.txt :language: text Now, let's create a ``.env`` file in the directory next to ``requirements.txt``. Add fill it in with the development env vars. For simplicity, set ``GITHUB_PRIVATE_KEY_PATH`` var and run the bash one-liner as shown in the example below. It turns a multiline private key file contents into a properly escaped single-line string. .. code:: touch .env GITHUB_PRIVATE_KEY_PATH=~/Downloads/your-app-slug.2019-03-24.private-key.pem cat $GITHUB_PRIVATE_KEY_PATH | python3.7 -c 'import sys; inline_private_key=r"\n".join(map(str.strip, sys.stdin.readlines())); print(f"GITHUB_PRIVATE_KEY='"'"'{inline_private_key}'"'"'", end="")' >> .env Now, copy-paste the *App ID* from the General App Settings page, which is still open in your browser and put it into `.env` file as a value for `GITHUB_APP_IDENTIFIER` variable. Also put there `DEBUG=true` and `ENV=dev` ``.env`` should look like this now: .. literalinclude:: resources/github-bot/.env After that, create a ``.gitignore`` in the same folder, it should contain ``.env`` entry. You can use the following command to download appropriate template:: wget -O - https://www.gitignore.io/api/git%2Cdotenv%2Clinux%2Cpydev%2Cpython%2Cwindows%2Cpycharm%2Ball%2Cjupyternotebooks%2Cvim%2Cwebstorm%2Cemacs >> .gitignore In the same directory, create another directory called ``github_bot``. Inside this new directory, create ``__main__.py``. Your ``github-bot/`` should now look as follows:: /github-bot /github-bot/.env /github-bot/.gitignore /github-bot/requirements.txt /github-bot/github_bot/__main__.py We'll start by creating a simple octomachinery app in ``__main__.py``. Edit ``__main__.py`` as follows: .. literalinclude:: resources/github-bot/github_bot/__main__.py :language: python :lines: 7,217- Save the file. Your webserver is now ready. From the command line and at the root of your project, enter the following:: python3 -m github_bot You should now see the following output:: DEBUG:octomachinery.app.server.runner:================ App version: 1.0.0 ================= DEBUG:asyncio:Using selector: EpollSelector DEBUG:octomachinery.app.server.machinery:The GitHub App env is set to `dev` INFO:octomachinery.app.server.machinery:Webhook secret is [NOT SET]: SIGNED WEBHOOKS WILL BE REJECTED INFO:octomachinery.app.server.machinery:Starting the following GitHub App: INFO:octomachinery.app.server.machinery:* app id: 21717 INFO:octomachinery.app.server.machinery:* private key SHA-1 fingerprint: 7d:96:e8:e5:8f:07:b5:10:97:85:2a:f4:33:72:b7:08:a5:81:82:92 INFO:octomachinery.app.server.machinery:* user agent: PyCon-Bot-by-webknjaz/1.0.0 (+https://github.com/apps/pyyyyyycoooon-booooot111) INFO:octomachinery.github.api.app_client:This GitHub App is installed into: INFO:octomachinery.github.api.app_client:* Installation id 491111 (installed to webknjaz) INFO:octomachinery.app.server.machinery:================= Serving on http://localhost:8080 ================== .. warning:: If you see some configuration error about invalid value of a setting, try checking env vars exported in your current terminal session. The dotenv library (``envparse``) used in the framework doesn't substitute those vars with values from ``.env`` file if they already exist in your env. You may need to ``unset`` them before proceeding. Open your browser and point it to http://localhost:8080. Alternatively, you can open another terminal and type:: curl -X GET localhost:8080 Whichever method you choose, you should see the output: "405: Method Not Allowed". That's expected: since the GitHub Apps event receiver is only supposed to process HTTP POST requests, other methods are not allowed. Update the Config Variables in Heroku ------------------------------------- Almost ready to actually start writing bots! Are you still on the Heroku dashboard? We are not done there just yet :) Go to the **Settings** tab. Click on the **Reveal Config Vars** button. Add three config variables here. The first one called **GITHUB_APP_IDENTIFIER**. Copy it from ``.env`` file you've created earlier. The next one is called **GITHUB_PRIVATE_KEY**. Copy it directly from the private key file you've downloaded earlier. No conversion is needed this time. Finally, set **HOST** to ``0.0.0.0``, ``DEBUG=false`` and ``ENV=prod``. Deploy to Heroku ---------------- Before we go further, let's first get that webservice deployed to Heroku. At the root of your project, create a new file called ``Procfile``, (without any extension). This file tells Heroku how it should run your app. Inside ``Procfile``:: web: python3 -m github_bot This will tell Heroku to run a web dyno using the command ``python3 -m github_bot``. Additionally, create ``runtime.txt`` file next to it containing:: python-3.7.2 This ensures that Heroku will provide Python 3.7 for us. Just as we need! 🎉 Your file structure should now look like the following:: /github-bot /github-bot/.env /github-bot/.gitignore /github-bot/requirements.txt /github-bot/runtime.txt /github-bot/Procfile /github-bot/github_bot/__main__.py Commit everything (except for ``.env`` file!) and push to GitHub. Open Heroku app dashboard (it may still be open somewhere among your browser tabs). Go to the "Deploy" tab. Under "Deployment method", choose GitHub. Connect your GitHub account if you haven't done that. Under "Search for a repository to connect to", enter your project name, e.g "github-bot". Press "Search". Once it found the right repo, press "Connect". Scroll down. Under Deploy a GitHub branch, choose "master", and click "Deploy Branch". (Optionally, enable automatic deployments) Watch the build log, and wait until it finished. When you see "Your app was successfully deployed", click on the "View" button. You should see "405: Method Not Allowed" (just as it was locally). Tip: Install Heroku toolbelt to see your logs. Once you have Heroku toolbelt installed, you can read the logs by:: heroku logs -a Pro tip: Install `Timber.io Logging `_ addon or similar to have a nicer view to more logs right in your browser. Your first GitHub bot! ---------------------- Ok NOW everything is finally ready. Let's start with something simple. Let's have a bot that **responds to every newly created issue in your project**. For example, whenever someone creates an issue, the bot will automatically say something like: "Thanks for the report, @user. I will look into this ASAP!" Go to the ``__main__.py`` file, in your ``github_bot`` codebase. The first change the part where we did is to add the following imports: .. literalinclude:: resources/github-bot/github_bot/__main__.py :language: python :lines: 8-10 Add the following coroutine (above **if __name__ == "__main__":**): .. literalinclude:: resources/github-bot/github_bot/__main__.py :language: python :lines: 16-20 This is where we are essentially subscribing to the GitHub ``issues`` event, and specifically to the "opened" issues event. ``@process_webhook_payload`` decorator automagically "unpacks" the event payload fields into the function arguments. ``github_api`` is a GitHub API client wrapper, which we've used in the previous section to make API calls to GitHub. Here, we get it from the contextvar proxy context, offered by octomachinery under the hood. This client is authorized against the installation bound to the current incoming event. .. _greet_author: Leave a comment whenever an issue is opened ''''''''''''''''''''''''''''''''''''''''''' Back to the task at hand. We want to *leave a comment whenever someone opened an issue*. Now that we're subscribed to the event, all we have to do now is to actually create the comment. We've done this in the previous section on the command line. You will recall the code is something like the following:: await github_api.post(url, data={"body": message}) Let's think about the ``url`` in this case. Previously, you might have constructed the url manually as follows:: url = f"/repos/mariatta/strange-relationship/issues/{issue_number}/comments" When we receive the webhook event however, the issue comment url is actually supplied in the payload. Take a look at GitHub's issue event payload `example (scroll it a bit) `_. It's a big JSON object. The portion we're interested in is:: { "action": "opened", "issue": { "url": ..., "comments_url": "https://api.github.com/repos/baxterthehacker/public-repo/issues/2/comments", "events_url": "...", "html_url": "...", ... } Notice that ``["issue"]["comments_url"]`` is actually the URL for posting comments to this particular issue. With this knowledge, your url is now: .. literalinclude:: resources/github-bot/github_bot/__main__.py :language: python :lines: 22 :dedent: 4 The next piece we want to figure out is what should the comment message be. For this exercise, we want to greet the author, and say something like "Thanks @author!". Take a look again at the issue event payload:: { "action": "opened", "issue": { "url": "...", ... "user": { "login": "baxterthehacker", "id": ..., ... } Did you spot it? The author's username can be accessed by ``issue["user"]["login"]``. So now your comment message should be: .. literalinclude:: resources/github-bot/github_bot/__main__.py :language: python :lines: 23-27 :dedent: 4 Piece all of that together, and actually make the API call to GitHub to create the comment: .. literalinclude:: resources/github-bot/github_bot/__main__.py :language: python :lines: 16-28 :emphasize-lines: 13 Your entire **__main__.py** should look like the following: .. literalinclude:: resources/github-bot/github_bot/__main__.py :language: python :lines: 7-12,16-28,217- Commit that file, push it to GitHub, and deploy it in Heroku. Almost there! Go to "Install App" tab in the GitHub App settings and install it into your test repo from there. It's needed so that your bot would start actually receiving events from that repository. Try and create an issue in the repo. See your bot in action!! Congrats! You now have a bot in place! Let's give it another job. .. _say_thanks: Say thanks when a PR has been merged '''''''''''''''''''''''''''''''''''' Let's now have the bot **say thanks, whenever a pull request has been merged**. For this case, you'll want to subscribe to the ``pull_request`` event, specifically when the ``action`` to the event is ``closed``. For reference, the relevant GitHub API documentation for the ``pull_request`` event is here: https://developer.github.com/v3/activity/events/types/#pullrequestevent. Scroll a bit to see the example payload for this event. Try it on your own. **Note**: A pull request can be closed without it getting merged. You'll need a way to determine whether the pull request was merged, or simply closed. .. _react_to_comments: React to issue comments ''''''''''''''''''''''' Everyone has opinion on the internet. Encourage more discussion by **automatically leaving a thumbs up reaction** for every comments in the issue. Ok you might not want to actually do that, (and whether it can actually encourage more discussion is questionable). Still, this can be a fun exercise. How about if the bot always gives **you** a thumbs up? Try it out on your own. - The relevant documentation is here: https://developer.github.com/v3/activity/events/types/#issuecommentevent - The example payload for the event is next to it - The API documentation for reacting to an issue comment is here: https://developer.github.com/v3/reactions/#create-reaction-for-an-issue-comment .. _label_prs: Label the pull request '''''''''''''''''''''' Let's make your bot do even more hard work. **Each time someone opens a pull request, have it automatically apply a label**. This can be a "pending review" or "needs review" label. The relevant API call is this: https://developer.github.com/v3/issues/#edit-an-issue .. _`profile Settings`: https://github.com/settings/profile .. _`Developer Settings`: https://github.com/settings/developers .. _`GitHub Apps`: https://github.com/settings/apps .. _`New GitHub App`: https://github.com/settings/apps/new .. _`Create a new GitHub App`: https://developer.github.com/apps/building-github-apps/creating-a-github-app/#creating-a-github-app