Categories: geek » emacs » org

RSS - Atom - Subscribe via email

Automatically refiling Org Mode headings based on tags

| org, emacs

I have lots of different things in my Org Mode inbox. Following the PARA method, I want to file them under projects, areas, resources, or archive so that I can find related things later. Actually, no, I don't want to refile them. I do want to be able to:

  • find all the pieces related to something when I'm ready to start working on a task
  • find useful links again, especially if I can use my own words

Refiling is annoying on my phone, so I tend to wait until I'm back at my computer. But even with org-refile-use-outline-path set to file and the ability to specify substrings, there's still a bit of friction.

Tagging is a little easier to do on my phone. I can add a few tags when I share a webpage or create a task.

I thought it would be nice to have something that automatically refiles my inbox headings tagged with various tags to other subtrees where I've set a :TAG_TARGET: property or something like that. For example, I can set the TAG_TARGET property to emacsconf to mean that anything tagged with :emacsconf: should get filed under there.

https://emacs.stackexchange.com/questions/36360/recursively-refiling-all-subtrees-with-tag-to-a-destination-org-mode

(defcustom my-org-refile-to-ids nil
  "Searches and IDs."
  :group 'sacha
  :type '(repeat (cons string string)))

(defun my-org-update-tag-targets ()
  (interactive)
  (setq my-org-refile-to-ids
        (let (list)
          (org-map-entries
           (lambda ()
             (cons (concat "+" (org-entry-get (point) "TAG_TARGET"))
                   (org-id-get-create)))
           "TAG_TARGET={.}" 'agenda)))
  (customize-save-variable 'my-org-refile-to-ids my-org-refile-to-ids))

(defun my-org-add-tag-target (tag)
  (interactive "MTag: ")
  (org-entry-put (point) "TAG_TARGET" tag)
  (push (cons (concat "+" tag) (org-id-get-create)) my-org-refile-to-ids)
  (customize-save-variable 'my-org-refile-to-ids my-org-refile-to-ids))

;; Based on https://emacs.stackexchange.com/questions/36360/recursively-refiling-all-subtrees-with-tag-to-a-destination-org-mode
(defun my-org-refile-matches-to-heading (match target-heading-id &optional scope copy)
  "Refile all headings within SCOPE (per `org-map-entries') to TARGET-HEADING-ID."
  (if-let (target-marker (org-id-find target-heading-id t))
      (let* ((target-rfloc (with-current-buffer (marker-buffer target-marker)
                             (goto-char target-marker)
                             (list (org-get-heading)
                                   (buffer-file-name (marker-buffer target-marker))
                                   nil
                                   target-marker)))
             (headings-to-copy (org-map-entries (lambda () (point-marker)) match scope)))
        (mapc
         (lambda (heading-marker)
           (with-current-buffer (marker-buffer heading-marker)
             (goto-char heading-marker)
             (org-refile nil nil target-rfloc (when copy "Copy"))))
         (nreverse headings-to-copy))
        (message "%s %d headings!"
                 (if copy "Copied" "Refiled")
                 (length headings-to-copy)))
    (warn "Could not find target heading %S" target-heading-id)))

(defun my-org-refile-to-tag-targets ()
  (interactive)
  (dolist (rule my-org-refile-to-ids)
    (my-org-refile-matches-to-heading (car rule) (cdr rule))))

So when I'm ready, I can call my-org-refile-to-tag-targets and have lots of things disappear from my inbox.

Next step might be to write a function that will refile just the current subtree (either going straight to the tag target or prompting me for a destination if there isn't a matching one), so I can look at stuff, decide if it needs to be scheduled first or something like that, and then send it somewhere. There must be something I can pass a property match to and it'll tell me if it matches the current subtree - probably something along the lines of org-make-tags-matcher

Anyway, just wanted to share this!

This is part of my Emacs configuration.

#EmacsConf backstage: automatically updating talk status from the crontab

| emacs, emacsconf, org

Now that I've figured out how to automatically play EmacsConf talks with crontab, I want to update our approach to using TRAMP and timers to run two tracks semi-automatically.

When a talk starts playing, I would like to:

  • announce it on IRC so that the talk details split up the chat log and make it easier to read
  • publish the media files to https://media.emacsconf.org/2023 (currently password-protected while testing) so that people can view them
  • update the talk wiki page with the media files and the captions
  • update the Youtube and Toobnix videos so that they're public: this is manual for the moment, since I haven't put time into automating it yet

I have code for most of this, and we've done it successfully for the past couple of years. I just need to refamiliarize myself with how to do it and how to set it up for testing during the dry run, and modify it to work with the new crontab-based system.

Setting things up

Last year, I set up the server with the ability to act as the controller, so that it wasn't limited to my laptop. The organizers notebook says I put it in the orga@ user's account, and I probably ran it under a screen session. I've submitted a talk for this year's conference, so I can use a test video for that one. First, I need to update the Ansible configuration for publishing and editing:

ansible-playbook -i inventory.yml prod-playbook.yml --tags publish,edit

I needed to add a version attribute to the git repo checkout in the Ansible playbook, since we'd switched from master to main. I also needed to set emacs_version to 29.1 since I started using seq-keep in my Emacs Lisp functions. For testing, I set emacsconf-publishing-phase to conference.

Act on TODO state changes

org-after-todo-state-change-hook makes it easy to automatically run functions when the TODO state changes. I add this hook that runs a list of functions and passes the talk information so that I don't have to parse the talk info in each function.

emacsconf-org-after-todo-state-change: Run all the hooks in ‘emacsconf-todo-hooks’.
(defun emacsconf-org-after-todo-state-change ()
  "Run all the hooks in `emacsconf-todo-hooks'.
If an `emacsconf-todo-hooks' entry is a list, run it only for the
tracks with the ID in the cdr of that list."
  (let* ((talk (emacsconf-get-talk-info-for-subtree))
         (track (emacsconf-get-track (plist-get talk :track))))
    (mapc
     (lambda (hook-entry)
       (cond
        ((symbolp hook-entry) (funcall hook-entry talk))
        ((member (plist-get track :id) (cdr hook-entry))
         (funcall (car hook-entry) talk))))
     emacsconf-todo-hooks)))

This can be enabled and disabled with the following functions.

emacsconf-add-org-after-todo-state-change-hook: Add FUNC to ‘org-after-todo-stage-change-hook’.
(defun emacsconf-add-org-after-todo-state-change-hook ()
  "Add FUNC to `org-after-todo-stage-change-hook'."
  (interactive)
  (with-current-buffer (find-buffer-visiting emacsconf-org-file)
    (add-hook 'org-after-todo-state-change-hook #'emacsconf-org-after-todo-state-change nil t)))

emacsconf-remove-org-after-todo-state-change-hook: Remove FUNC from ‘org-after-todo-stage-change-hook’.
(defun emacsconf-remove-org-after-todo-state-change-hook ()
  "Remove FUNC from `org-after-todo-stage-change-hook'."
  (interactive)
  (with-current-buffer (find-buffer-visiting emacsconf-org-file)
    (remove-hook 'org-after-todo-state-change-hook
                 #'emacsconf-org-after-todo-state-change  t)))

Announce on IRC

This is still much the same as last year.

emacsconf-erc-announce-on-change: Announce talk.
(defun emacsconf-erc-announce-on-change (talk)
  "Announce talk."
  (let ((func
         (pcase org-state
           ("PLAYING" #'erc-cmd-NOWPLAYING)
           ("CLOSED_Q" #'erc-cmd-NOWCLOSEDQ)
           ("OPEN_Q" #'erc-cmd-NOWOPENQ)
           ("UNSTREAMED_Q" #'erc-cmd-NOWUNSTREAMEDQ)
           ("TO_ARCHIVE" #'erc-cmd-NOWDONE))))
    (when func
      (funcall func talk))))

Here's a sample command that announces that the talk is now playing.

erc-cmd-NOWPLAYING: Set the channel topics to announce TALK.
(defun erc-cmd-NOWPLAYING (talk)
  "Set the channel topics to announce TALK."
  (interactive (list (emacsconf-complete-talk-info)))
  (when (stringp talk)
    (setq talk (or (emacsconf-find-talk-info talk) (error "Could not find talk %s" talk))))
  ;; Announce it in the track's channel
  (if (emacsconf-erc-recently-announced (format "---- %s:" (plist-get talk :slug)))
      (message "Recently announced, skipping")
    (when (plist-get talk :track)
      (emacsconf-erc-with-channels (list (concat "#" (plist-get talk :channel)))
        (erc-cmd-TOPIC
         (format
          "%s: %s (%s) pad: %s Q&A: %s | %s"
          (plist-get talk :slug)
          (plist-get talk :title)
          (plist-get talk :speakers)
          (plist-get talk :pad-url)
          (plist-get talk :qa-info)
          (car (assoc-default
                (concat "#" (plist-get talk :channel))
                emacsconf-topic-templates))))
        (erc-send-message (format "---- %s: %s - %s ----"
                                  (plist-get talk :slug)
                                  (plist-get talk :title)
                                  (plist-get talk :speakers-with-pronouns)))
        (erc-send-message
         (concat "Add your notes/questions to the pad: " (plist-get talk :pad-url)))
        (cond
         ((string-match "live" (or (plist-get talk :q-and-a) ""))
          (erc-send-message (concat "Live Q&A: " (plist-get talk :bbb-redirect))))
         ((plist-get talk :irc)
          (erc-send-message (format "or discuss the talk on IRC (nick: %s)"
                                    (plist-get talk :irc)))))))
    ;; Short announcement in #emacsconf
    (emacsconf-erc-with-channels (list emacsconf-erc-hallway emacsconf-erc-org)
      (erc-send-message (format "-- %s track: %s: %s (watch: %s, pad: %s, channel: #%s)"
                                (plist-get talk :track)
                                (plist-get talk :slug)
                                (plist-get talk :title)
                                (plist-get talk :watch-url)
                                (plist-get talk :pad-url)
                                (plist-get talk :channel))))))

Because the commands change the topic, I need to op the user in all the Emacsconf channels.

erc-cmd-OPALL
(defun erc-cmd-OPALL (&optional nick)
  (emacsconf-erc-with-channels (mapcar 'car emacsconf-topic-templates)
    (if nick
        (erc-cmd-OP nick)
      (erc-cmd-OPME))))

This code is in emacsconf-erc.el.

Publish media files

We used to scramble to upload all the videos in the days or weeks after the conference, since the presentations were live. Since we switched to encouraging speakers to upload videos before the conference, we've been able to release the videos pretty much as soon as the talk starts playing. This code in emacsconf-publish.el takes care of copying the files from the backstage to the public media directory, republishing the index, and republishing the playlist. That way, people who come in late or who want to refer to the video can easily get the full video right away.

emacsconf-publish-media-files-on-change: Publish the files and update the index.
(defun emacsconf-publish-media-files-on-change (talk)
  "Publish the files and update the index."
  (interactive (list (emacsconf-complete-talk-info)))
  (let ((org-state (if (boundp 'org-state) org-state (plist-get talk :status))))
    (if (plist-get talk :public)
        ;; Copy main extension files from backstage to public
        (let ((files (directory-files emacsconf-backstage-dir nil
                                      (concat "^"
                                              (regexp-quote (plist-get talk :file-prefix))
                                              (regexp-opt emacsconf-main-extensions)))))
          (mapc (lambda (file)
                  (when (and
                         (not (file-exists-p (expand-file-name file emacsconf-public-media-directory)))
                         (or (not (string-match "--main.vtt$" file))
                             (plist-get talk :captions-edited)))
                    (copy-file (expand-file-name file emacsconf-backstage-dir)
                               (expand-file-name file emacsconf-public-media-directory) t)))
                files))
      ;; Remove files from public
      (let ((files (directory-files emacsconf-public-media-directory nil
                                    (concat "^"
                                            (regexp-quote (plist-get talk :file-prefix)
                                                          )))))
        (mapc (lambda (file)
                (delete-file (expand-file-name file emacsconf-public-media-directory)))
              files)))
    (emacsconf-publish-public-index)
    (emacsconf-publish-playlist
     (expand-file-name "index.m3u" emacsconf-public-media-directory)
     (concat emacsconf-name " " emacsconf-year)
     (emacsconf-public-talks (emacsconf-get-talk-info)))))

The :public property is automatically added by this function based on the TODO status or the time:

emacsconf-add-talk-status: Add status label and public info.
(defun emacsconf-add-talk-status (o)
  "Add status label and public info."
  (plist-put o :status-label
             (or (assoc-default (plist-get o :status)
                                emacsconf-status-types 'string= "")
                 (plist-get o :status)))
  (when (or
         (member (plist-get o :status)
                 (split-string "PLAYING CLOSED_Q OPEN_Q UNSTREAMED_Q TO_ARCHIVE TO_EXTRACT TO_FOLLOW_UP DONE"))
         (time-less-p (plist-get o :start-time)
                      (current-time)))
    (plist-put o :public t))
  o)

Update the wiki page

This function updates the schedule page and the page for the talk. It's also in emacsconf-publish.el.

emacsconf-publish-update-talk: Publish the schedule page and the page for this talk.
(defun emacsconf-publish-update-talk (talk)
  "Publish the schedule page and the page for this talk."
  (interactive (list (emacsconf-complete-talk-info)))
  (when (stringp talk) (setq talk (emacsconf-resolve-talk talk)))
  (when (functionp 'emacsconf-upcoming-insert-or-update)
    (emacsconf-upcoming-insert-or-update))
  (emacsconf-publish-with-wiki-change
    (let ((info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
      (emacsconf-publish-before-page talk info)
      (emacsconf-publish-after-page talk info)
      (emacsconf-publish-schedule info))))

It uses the publishing functions I described in this post on adding talks to the wiki.

This macro commits changes when emacsconf-publish-autocommit-wiki is t, so I need to set that also.

emacsconf-publish-with-wiki-change
(defmacro emacsconf-publish-with-wiki-change (&rest body)
  (declare (indent 0) (debug t))
  `(progn
     ,@body
     (emacsconf-publish-commit-and-push-wiki-maybe
      ,emacsconf-publish-autocommit-wiki
      (and (stringp ,(car body)) ,(car body)))))

Make the videos public on YouTube and Toobnix

This is low-priority, but it might be nice to figure out. The easiest way is probably to use open the Youtube/Toobnix URLs on my computer and then use either Tampermonkey or Spookfox to set the talk to public. Someday!

Update the talk status on the server

Last year, I experimented with having the shell scripts automatically update the status of the talk from TO_STREAM to PLAYING and from PLAYING to CLOSED_Q. Since I've moved the talk-running into track-specific crontabs, now I need to sudo back to the orga user and set XDG_RUNTIME_DIR in order to use emacsclient. I can call this with sudo -u orga talk $slug $status in the roles/obs/templates/handle-session script.

Here's the Ansible template for roles/prerec/templates/talk. It uses getent to look up the user ID.

#!/bin/bash
# 
# How to use: talk slug from-status-regexp to-status
# or talk slug to-status

SLUG="$1"
FROM_STATUS="$2"
TO_STATUS="$3"
XDG_RUNTIME_DIR=/run/user/

if [ "x$TO_STATUS" == "x" ]; then
    FROM_STATUS=.
    TO_STATUS="$2"
fi
cd 
#echo "Pulling conf.org..."
#git pull
echo "Updating status..."
XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR emacsclient --eval "(emacsconf-with-todo-hooks (emacsconf-update-talk-status \"$SLUG\" \"$FROM_STATUS\" \"$TO_STATUS\"))" -a emacs
#echo "Committing and pushing in the background"
#git commit -m "Update task status for $SLUG from $FROM_STATUS to $TO_STATUS" conf.org
#git push &

Testing notes

Looks like everything works fine when I run it from the crontab: the talk status is updated, the media files are published, the wiki is updated, and the talks are announced on IRC. Backup plan A is to manually control the talk status using Emacs on the server. Backup plan B is to control the talk status using Emacs on my laptop. Backup plan C is to call the individual functions instead of relying on the todo state change functions. I think it'll all work out, although I'll probably want to do another dry run at some point to make sure. Slowly getting there…

Getting Mermaid JS and ob-mermaid running on my system - needed to symlink Chromium for Puppeteer

| emacsconf, nodejs, org, emacs

I wanted to use Mermaid to make diagrams, but I ran into this issue when trying to run it:

Error: Could not find Chromium (rev. 1108766). This can occur if either
 1. you did not perform an installation before running the script (e.g. `npm install`) or
 2. your cache path is incorrectly configured (which is: /home/sacha/.cache/puppeteer).
For (2), check out our guide on configuring puppeteer at https://pptr.dev/guides/configuration.

It turns out that I needed to do the following:

sudo npm install -g puppeteer mermaid @mermaid-js/mermaid-cli --unsafe-perm
# Cache Chromium for my own user
node /usr/lib/node_modules/puppeteer/install.js --unsafe-perm
sudo npm install -g mermaid @mermaid-js/mermaid-cli
ln -s ~/.cache/puppeteer/chrome/linux-117.0.5938.149 ~/.cache/puppeteer/chrome/linux-1108766
ln -s ~/.cache/puppeteer/chrome/linux-117.0.5938.149/chrome-linux64 ~/.cache/puppeteer/chrome/linux-117.0.5938.149/chrome-linux

(The exact versions might be different for your installation.)

Then I could make a Mermaid file and try it out with mmdc -i input.mmd -o output.svg, and then I could confirm that it works directly from Org with ob-mermaid:

sequenceDiagram
    input ->> res: original.mp4;
    res ->> backstage: reencoded.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    input ->> res: main.vtt;
    res ->> backstage: main.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    input ->> res: normalized.opus;
    res ->> backstage: main.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    backstage ->> media: public files;
media-flow.png

#EmacsConf backstage: Using Spookfox to automate creating BigBlueButton rooms in Mozilla Firefox

| emacsconf, emacs, org

Naming conventions make it easier for other people to find things. Just like with file prefixes, I like to use a standard naming pattern for our BigBlueButton web conference rooms. For EmacsConf 2022, we used ec22-day-am-gen Speaker name (slugs). For EmacsConf 2023, I want to set up the BigBlueButton rooms before the schedule settles down, so I won't encode the time or track information into it. Instead, I'll use Speaker name (slugs) - emacsconf2023.

BigBlueButton does have an API for managing rooms, but that requires a shared secret that I don't know yet. I figured I'd just automate it through my browser. Over the last year, I've started using Spookfox to control the Firefox web browser from Emacs. It's been pretty handy for scrolling webpages up and down, so I wondered if I could replace my old xdotool-based automation. Here's what I came up with for this year.

First, I need a function that creates the BBB room for a group of talks and updates the Org entry with the URL. Adding a slight delay makes it a bit more reliable.

emacsconf-spookfox-create-bbb: Create a BBB room for this group of talks.
(defun emacsconf-spookfox-create-bbb (group)
  "Create a BBB room for this group of talks.
GROUP is (email . (talk talk talk)).
Needs a Spookfox connection."
  (let* ((bbb-name
          (format "%s (%s) - %s%s"
                  (mapconcat (lambda (o) (plist-get o :slug)) (cdr group) ", ")
                  (plist-get (cadr group) :speakers)
                  emacsconf-id
                  emacsconf-year))
         path
         (retrieve-command
          (format
           "window.location.origin + [...document.querySelectorAll('h4.room-name-text')].find((o) => o.textContent.trim() == '%s').closest('tr').querySelector('.delete-room').getAttribute('data-path')"
           bbb-name))
         (create-command (format "document.querySelector('#create-room-block').click();
document.querySelector('#create-room-name').value = \"%s\";
document.querySelector('#room_mute_on_join').click();
document.querySelector('.create-room-button').click();"
                                 bbb-name)))
    (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t))
    (unless path
      (dolist (cmd (split-string create-command ";"))
        (spookfox-js-injection-eval-in-active-tab cmd t)
        (sleep-for 2))
      (sleep-for 2)
      (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t)))
    (when path
      (dolist (talk (cdr group))
        (save-window-excursion
          (emacsconf-with-talk-heading talk
            (org-entry-put (point) "ROOM" path))))
      (cons bbb-name path))))

Then I need to iterate over the list of talks that have live Q&A sessions but don't have BBB rooms assigned yet so that I can create them.

emacsconf-spookfox-create-bbb-for-live-talks: Create BBB rooms for talks that don’t have them yet.
(defun emacsconf-spookfox-create-bbb-for-live-talks ()
  "Create BBB rooms for talks that don't have them yet."
  (let* ((talks (seq-filter
                 (lambda (o)
                   (and (string-match "live" (or (plist-get o :q-and-a) ""))
                        (not (string= (plist-get o :status) "CANCELLED"))
                        (not (plist-get o :bbb-room))))
                 (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
         (groups (and talks (emacsconf-mail-groups talks))))
    (dolist (group groups)
      (emacsconf-spookfox-create-bbb group))))

The result: a whole bunch of rooms ready for people to check in.

2023-10-14_09-24-34.png
Figure 1: BigBlueButton rooms

Using Spookfox to communicate with Firefox from Emacs Lisp made it easy to get data in and out of my browser. Handy!

This code is in emacsconf-spookfox.el.

#EmacsConf backstage: file prefixes

| emacs, org, emacsconf

Sometimes it makes sense to dynamically generate information related to a talk and then save it as an Org property so that I can manually edit it. For example, we like to name all the talk files using this pattern: "emacsconf-year-slug--title--speakers". That's a lot to type consistently! We can generate most of these prefixes automatically, but some might need tweaking, like when the talk title or speaker names have special characters.

Calculating the file prefix for a talk

First we need something that turns a string into an ID.

emacsconf-slugify: Turn S into an ID.
(defun emacsconf-slugify (s)
  "Turn S into an ID.
Replace spaces with dashes, remove non-alphanumeric characters,
and downcase the string."
  (replace-regexp-in-string
   " +" "-"
   (replace-regexp-in-string
    "[^a-z0-9 ]" ""
    (downcase s))))

Then we can use that to calculate the file prefix for a given talk.

emacsconf-file-prefix: Create the file prefix for TALK
(defun emacsconf-file-prefix (talk)
  "Create the file prefix for TALK."
  (concat emacsconf-id "-"
          emacsconf-year "-"
          (plist-get talk :slug) "--"
          (emacsconf-slugify (plist-get talk :title))
          (if (plist-get talk :speakers)
              (concat "--"
                     (emacsconf-slugify (plist-get talk :speakers)))
            "")))

Then we can map over all the talk entries that don't have FILE_PREFIX defined:

emacsconf-set-file-prefixes: Set the FILE_PREFIX property for each talk entry that needs it.
(defun emacsconf-set-file-prefixes ()
  "Set the FILE_PREFIX property for each talk entry that needs it."
  (interactive)
  (org-map-entries
   (lambda ()
     (org-entry-put
      (point) "FILE_PREFIX"
      (emacsconf-file-prefix (emacsconf-get-talk-info-for-subtree))))
   "SLUG={.}-FILE_PREFIX={.}"))

That stores the file prefix in an Org property, so we can edit it if it needs tweaking.

output-2023-10-10-15:17:17.gif
Figure 1: Setting the FILE_PREFIX for all talks that don't have that yet

Renaming files to match the file prefix

Now that we have that, how can we use it? One way is to rename files from within Emacs. I can mark multiple files with Dired's m command or work on them one at a time. If there are several files with the same extension, I can specify something to add to the filename to tell them apart.

emacsconf-rename-files: Rename the marked files or the current file to match TALK.
(defun emacsconf-rename-files (talk &optional filename)
  "Rename the marked files or the current file to match TALK.
If FILENAME is specified, use that as the extra part of the filename after the prefix.
This is useful for distinguishing files with the same extension.
Return the list of new filenames."
  (interactive (list (emacsconf-complete-talk-info)))
  (prog1
      (mapcar
       (lambda (file)
         (let* ((extra
                 (or filename
                     (read-string (format "Filename (%s): " (file-name-base file)))))
                (new-filename
                 (expand-file-name
                  (concat (plist-get talk :file-prefix)
                          (if (string= extra "")
                              ""
                            (concat "--" extra))
                          "."
                          (file-name-extension file))
                  (file-name-directory file))))
           (rename-file file new-filename t)
           new-filename))
       (or (dired-get-marked-files) (list (buffer-file-name))))
    (when (derived-mode-p 'dired-mode)
      (revert-buffer))))

output-2023-10-10-14:18:34.gif
Figure 2: Renaming multiple files

Working with files on other computers

Because Dired works over TRAMP, I can use that to rename files on a remote server without changing anything about the code. I can open the remote directory with Dired and everything just works.

TRAMP also makes it easy to copy a file to the backstage directory after it's renamed, which saves me having to do that as a separate step.

emacsconf-rename-and-upload-to-backstage: Rename marked files or the current file, then upload to backstage.
(defun emacsconf-rename-and-upload-to-backstage (talk &optional filename)
  "Rename marked files or the current file, then upload to backstage."
  (interactive (list (emacsconf-complete-talk-info)))
  (mapc
   (lambda (file)
     (copy-file
      file
      (expand-file-name
       (file-name-nondirectory file)
       emacsconf-backstage-dir)
      t))
   (emacsconf-rename-files talk)))

So if my emacsconf-backstage-dir is set to /ssh:orga@res:/var/www/res.emacsconf.org/2023/backstage, then it looks up the details for res in my ~/.ssh/config and copies the file there.

Renaming files using information from a JSON

What if I don't want to rename the files from Emacs? If I use Emacs's JSON support to export some information from the talks as a JSON file, then I can easily use that data from the command line.

Here's how I export the talk information:

emacsconf-talks-json: Return JSON format with a subset of talk information.
(defun emacsconf-publish-talks-json ()
  "Return JSON format with a subset of talk information."
  (json-encode
   (list
    :talks
    (mapcar
     (lambda (o)
       (apply
        'list
        (cons :start-time (format-time-string "%FT%T%z" (plist-get o :start-time) t))
        (cons :end-time (format-time-string "%FT%T%z" (plist-get o :end-time) t))
        (mapcar
         (lambda (field)
           (cons field (plist-get o field)))
         '(:slug :title :speakers :pronouns :pronunciation :url :track :file-prefix))))
     (emacsconf-filter-talks (emacsconf-get-talk-info))))))

emacsconf-publish-talks-json-to-files
(defun emacsconf-publish-talks-json-to-files ()
  "Export talk information as JSON so that we can use it in shell scripts."
  (interactive)
  (mapc (lambda (dir)
          (when (and dir (file-directory-p dir))
            (with-temp-file (expand-file-name "talks.json" dir)
              (insert (emacsconf-talks-json)))))
        (list emacsconf-res-dir emacsconf-ansible-directory)))

Then I can use jq to extract the information with

jq -r '.talks[] | select(.slug=="'$SLUG'")["file-prefix"]' < $TALKS_JSON

Here it is in the context of a shell script that renames the given file to match a talk's prefix.

#!/bin/bash
# 
# Usage: rename-original.sh $slug $file [$extra] [$talks-json]
SLUG=$1
FILE=$2
TALKS_JSON=${4:-~/current/talks.json}
EXTRA=""
if [ -z ${3-unset} ]; then
    EXTRA=""
elif [ -n "$3" ]; then
    EXTRA="--$3"
elif echo "$FILE" | grep -e '\(webm\|mp4\|mov\)'; then
    EXTRA="--original"
fi
filename=$(basename -- "$FILE")
extension="${filename##*.}"
filename="${filename%.*}"
FILE_PREFIX=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["file-prefix"]' < $TALKS_JSON)
mv "$FILE" $FILE_PREFIX$EXTRA.$extension
echo $FILE_PREFIX$EXTRA.$extension
# Copy to original if needed
if [ -f $FILE_PREFIX--original.webm ] && [ ! -f $FILE_PREFIX--main.$extension ]; then
    cp $FILE_PREFIX--original.$extension $FILE_PREFIX--main.webm
fi

Then I can use something like rename-original.sh emacsconf video.webm to emacsconf-2023-emacsconf--emacsconforg-how-we-use-org-mode-and-tramp-to-organize-and-run-a-multitrack-conference--sacha-chua--original.webm.

Working with PsiTransfer-uploaded files

JSON support is useful for getting files into our system, too. For EmacsConf 2022, we used PsiTransfer as a password-protected web-based file upload service. That was much easier for speakers to deal with than FTP, especially for large files. PsiTransfer makes a JSON file for each batch of uploads, which is handy because the uploaded files are named based on the key instead of keeping their filenames and extensions. I wrote a function to copy an uploaded file from the PsiTransfer directory to the backstage directory, renaming it along the way. That meant that I could open the JSON for the uploaded files via TRAMP and then copy a file between two remote directories without manually downloading it to my computer.

emacsconf-upload-copy-from-json: Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.
(defun emacsconf-upload-copy-from-json (talk key filename)
  "Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.
The file is associated with TALK. KEY identifies the file in a multi-file upload.
FILENAME specifies an extra string to add to the file prefix if needed."
  (interactive (let-alist (json-parse-string (buffer-string) :object-type 'alist)
                 (list (emacsconf-complete-talk-info)
                       .metadata.key
                       (read-string (format "Filename: ")))))
  (let ((new-filename (concat (plist-get talk :file-prefix)
                              (if (string= filename "")
                                  filename
                                (concat "--" filename))
                              "."
                              (let-alist (json-parse-string (buffer-string) :object-type 'alist)
                                (file-name-extension .metadata.name)))))
    (copy-file
     key
     (expand-file-name new-filename emacsconf-backstage-dir)
     t)))

So that's how I can handle things that can be mostly automated but that might need a little human intervention: use Emacs Lisp to make a starting point, tweak it a little if needed, and then make it easy to use that value elsewhere. Renaming files can be tricky, so it's good to reduce the chance for typos!

#EmacsConf backstage: looking at EmacsConf's growth over 5 years, and how to do pivot tables and graphs with Org Mode and the Python pandas library

| emacsconf, emacs, org, python

Having helped organize EmacsConf for a number of years now, I know that I usually panic about whether we have submissions partway through the call for participation. This causes us to extend the CFP deadline and ask people to ask more people to submit things, and then we end up with a wonderful deluge of talks that I then have to somehow squeeze into a reasonable-looking schedule.

This year, I managed to not panic and I also resisted the urge to extend the CFP deadline, trusting that there will actually be tons of cool stuff. It helped that my schedule SVG code let me visualize what the conference could feel like with the submissions so far, so we started with a reasonably nice one-track conference and built up from there. It also helped that I'd gone back to the submissions for 2022 and plotted them by the number of weeks before the CFP deadline, and I knew that there'd be a big spike from all those people whose Org DEADLINE: properties would nudge them into finalizing their proposals.

Out of curiosity, I wanted to see how the stats for this year compared with previous years. I wrote a small function to collect the data that I wanted to summarize:

emacsconf-count-submissions-by-week: Count submissions in INFO by distance to CFP-DEADLINE.
(defun emacsconf-count-submissions-by-week (&optional info cfp-deadline)
  "Count submissions in INFO by distance to CFP-DEADLINE."
  (setq cfp-deadline (or cfp-deadline emacsconf-cfp-deadline))
  (setq info (or info (emacsconf-get-talk-info)))
  (cons '("Weeks to CFP end date" "Count" "Hours")
        (mapcar (lambda (entry)
                  (list (car entry)
                        (length (cdr entry))
                        (apply '+ (mapcar 'cdr (cdr entry)))))
                (seq-group-by
                 'car
                 (sort
                  (seq-keep
                   (lambda (o)
                     (and (emacsconf-publish-talk-p o)
                          (plist-get o :date-submitted)
                          (cons (floor (/ (days-between (plist-get o :date-submitted) cfp-deadline)
                                          7.0))
                                (string-to-number
                                 (or (plist-get o :video-duration)
                                     (plist-get o :time)
                                     "0")))))
                   info)
                  (lambda (a b) (< (car a) (car b))))))))

and then I ran it against the different files for each year, filling in the previous years' data as needed. The resulting table is pretty long, so I've put that in a collapsible section.

(let ((years `((2023 "~/proj/emacsconf/2023/private/conf.org" "2023-09-15")
               (2022 "~/proj/emacsconf/2022/private/conf.org" "2022-09-18")
               (2021 "~/proj/emacsconf/2021/private/conf.org" "2021-09-30")
               (2020 "~/proj/emacsconf/wiki/2020/submissions.org" "2020-09-30")
               (2019 "~/proj/emacsconf/2019/private/conf.org" "2019-08-31"))))
  (append
   '(("Weeks to CFP" "Year" "Count" "Minutes"))
   (seq-mapcat
    (lambda (year-info)
      (let ((emacsconf-org-file (elt year-info 1))
            (emacsconf-cfp-deadline (elt year-info 2))
            (year (car year-info)))
        (mapcar (lambda (o) (list (car o) year (cadr o) (elt o 2)))
                (cdr (emacsconf-count-submissions-by-week (emacsconf-get-talk-info) emacsconf-cfp-deadline)))))
    years)))
Table
Weeks to CFP Year Count Minutes
-12 2023 4 70
-9 2023 2 30
-7 2023 2 30
-5 2023 2 30
-4 2023 2 60
-3 2023 3 40
-2 2023 5 130
-1 2023 10 180
0 2023 8 140
1 2023 1 20
-8 2022 2 25
-5 2022 2 31
-3 2022 2 31
-2 2022 2 17
-1 2022 8 191
0 2022 8 110
1 2022 5 107
-8 2021 4 50
-7 2021 2 17
-6 2021 1 7
-5 2021 2 22
-4 2021 2 19
-3 2021 5 73
-2 2021 1 10
-1 2021 12 163
0 2021 13 197
1 2021 1 10
2 2021 1 10
-5 2020 1 10
-4 2020 1 15
-2 2020 1 30
-1 2020 4 68
0 2020 21 424
1 2020 7 152
-5 2019 2 45
-4 2019 1 21
-2 2019 6 126
-1 2019 9 82
0 2019 9 148
2 2019 1 7

Some talks were proposed off-list and are not captured here, and cancelled or withdrawn talks weren't included either. The times for previous years use the actual video time, and the times for this year use proposed times.

Off the top of my head, I didn't know of an easy way to make a pivot table or cross-tab using just Org Mode or Emacs Lisp. I tried using datamash, but I was having a hard time getting my output just the way I wanted it. Fortunately, it was super-easy to get my data from an Org table into Python so I could use pandas.pivot_table. Because I had used #+NAME: submissions-by-week to label the table, I could use :var data=submissions-by-week to refer to the data in my Python program. Then I could summarize them by week.

Here's the number of submissions by the number of weeks to the original CFP deadline, so we can see people generally like to target the CFP date.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True)
fig, ax = plt.subplots()
figure = df.plot(title='Number of submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('number-of-submissions.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 4
-9 0 0 0 0 2
-8 0 0 4 2 0
-7 0 0 2 0 2
-6 0 0 1 0 0
-5 2 1 2 2 2
-4 1 1 2 0 2
-3 0 0 5 2 3
-2 6 1 1 2 5
-1 9 4 12 8 10
0 9 21 13 8 8
1 0 7 1 5 1
2 1 0 1 0 0
number-of-submissions.png

Calculating the cumulative number of submissions might be more useful. Here, each row shows the number received so far.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax = plt.subplots()
figure = df.plot(title='Cumulative submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('cumulative-submissions.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 4
-9 0 0 0 0 6
-8 0 0 4 2 6
-7 0 0 6 2 8
-6 0 0 7 2 8
-5 2 1 9 4 10
-4 3 2 11 4 12
-3 3 2 16 6 15
-2 9 3 17 8 20
-1 18 7 29 16 30
0 27 28 42 24 38
1 27 35 43 29 39
2 28 35 44 29 39
cumulative-submissions.png
Figure 1: Cumulative submissions by number of weeks to CFP end date

And here's the cumulative number of minutes based on the proposals.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax = plt.subplots()
figure = df.plot(title='Cumulative minutes by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('cumulative-minutes.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 70
-9 0 0 0 0 100
-8 0 0 50 25 100
-7 0 0 67 25 130
-6 0 0 74 25 130
-5 45 10 96 56 160
-4 66 25 115 56 220
-3 66 25 188 87 260
-2 192 55 198 104 390
-1 274 123 361 295 570
0 422 547 558 405 710
1 422 699 568 512 730
2 429 699 578 512 730
cumulative-minutes.png
Figure 2: Cumulative minutes by number of weeks to the CFP end date

So… yeah… 730 minutes of talks for this year… I might've gotten a little carried away. But I like all the talks! And I want them to be captured in videos and maybe even transcribed by people who will take the time to change misrecognized words like Emax into Emacs! And I want people to be able to connect with other people who are interested in the sorts of stuff they're doing! So we're going to make it happen. The draft schedule's looking pretty full, but I think it'll work out, especially if the speakers send in their videos on time. Let's see how it all works out!

(…and look, I even got to learn how to do pivot tables and graphs with Python!)

Org protocol: following Org links from outside Emacs

| org, emacs

_xor had an interesting idea: can we use org-protocol to link to things inside Emacs, so that we can have a webpage with bookmarks into our Org files? Here's a quick hack that reuses org-store-link and org-link-open.

(defun org-protocol-open-link (info)
  "Process an org-protocol://open style url with INFO."
  (org-link-open (car (org-element-parse-secondary-string (plist-get info :link) '(link)))))

(defun org-protocol-copy-open-link (arg)
  (interactive "P")
  (kill-new (concat "org-protocol://open?link=" (url-hexify-string (org-store-link arg)))))

(with-eval-after-load 'org
  (add-to-list 'org-protocol-protocol-alist
               '("org-open" :protocol "open" :function org-protocol-open-link)))

To make exporting and following easier, we also need a little code to handle org-protocol links inside Org.

(defun org-protocol-follow (path &rest _)
  "Follow the org-protocol link for PATH."
  (org-protocol-check-filename-for-protocol (concat "org-protocol:" path) nil nil))

(defun org-protocol-export (path desc format info)
  "Export an org-protocol link."
  (setq path (concat "org-protocol:" path))
  (setq desc (or desc path))
  (pcase format
    (`html (format "<a href=\"%s\">%s</a>" path desc))
    (`11ty (format "<a href=\"%s\">%s</a>" path desc))
    (`latex (org-latex-link path desc info))
    (`ascii (org-ascii-link path desc info))
    (`md (org-md-link path desc info))
    (_ path)))

(with-eval-after-load 'org
  (org-link-set-parameters "org-protocol"
                           :follow #'org-protocol-follow
                           :export #'org-protocol-export))

Now I can use org-protocol-copy-open-link to copy a link to the current location, and I can put it into my Org files.

Example bare link to the Org manual, which will work only if you have open in the org-protocol-protocol-alist:

org-protocol://open?link=%5B%5Binfo%3Aorg%23Protocols%5D%5Borg%23Protocols%5D%5D

With a description:

Org manual - Protocols

This is part of my Emacs configuration.