Categories: geek

RSS - Atom - Subscribe via email

Re-encoding the EmacsConf videos with FFmpeg and GNU Parallel

| geek, linux, emacsconf

It turns out that using -crf 56 compressed the EmacsConf a little too aggressively, losing too much information in the video. We wanted to reencode everything, maybe going back to the default value of -crf 32. My laptop would have taken a long time to do all of those videos. Fortunately, one of the other volunteers shared a VM on a machine with 12 cores, and I had access to a few other systems. It was a good opportunity to learn how to use GNU Parallel to send jobs to different machines and retrieve the results.

First, I updated the compression script, compress-video-low.sh:

Q=$1
WIDTH=1280
HEIGHT=720
AUDIO_RATE=48000
VIDEO_FILTER="scale=w=${WIDTH}:h=${HEIGHT}:force_original_aspect_ratio=1,pad=${WIDTH}:${HEIGHT}:(ow-iw)/2:(oh-ih)/2,fps=25,colorspace=all=bt709:iall=bt601-6-625:fast=1"
FILE=$2
SUFFIX=$Q
shift
shift
ffmpeg -y -i "$FILE"  -pixel_format yuv420p -vf $VIDEO_FILTER -colorspace 1 -color_primaries 1 -color_trc 1 -c:v libvpx-vp9 -b:v 0 -crf $Q -aq-mode 2 -tile-columns 0 -tile-rows 0 -frame-parallel 0 -cpu-used 8 -auto-alt-ref 1 -lag-in-frames 25 -g 240 -pass 1 -f webm -an -threads 8 /dev/null &&
if [[ $FILE =~ "webm" ]]; then
    ffmpeg -y -i "$FILE" $*  -pixel_format yuv420p -vf $VIDEO_FILTER -colorspace 1 -color_primaries 1 -color_trc 1 -c:v libvpx-vp9 -b:v 0 -crf $Q -tile-columns 2 -tile-rows 2 -frame-parallel 0 -cpu-used -5 -auto-alt-ref 1 -lag-in-frames 25 -pass 2 -g 240 -ac 2 -threads 8 -c:a copy "${FILE%.*}--compressed$SUFFIX.webm"
else
    ffmpeg -y -i "$FILE" $*  -pixel_format yuv420p -vf $VIDEO_FILTER -colorspace 1 -color_primaries 1 -color_trc 1 -c:v libvpx-vp9 -b:v 0 -crf $Q -tile-columns 2 -tile-rows 2 -frame-parallel 0 -cpu-used -5 -auto-alt-ref 1 -lag-in-frames 25 -pass 2 -g 240 -ac 2 -threads 8 -c:a libvorbis "${FILE%.*}--compressed$SUFFIX.webm"
fi

I made an originals.txt file with all the original filenames. It looked like this:

emacsconf-2020-frownies--the-true-frownies-are-the-friends-we-made-along-the-way-an-anecdote-of-emacs-s-malleability--case-duckworth.mkv
emacsconf-2021-montessori--emacs-and-montessori-philosophy--grant-shangreaux.webm
emacsconf-2021-pattern--emacs-as-design-pattern-learning--greta-goetz.mp4
...

I set up a ~/.parallel/emacsconf profile with something like this so that I could use three computers and my laptop, sending one job each and displaying progress:

--sshlogin computer1 --sshlogin computer2 --sshlogin computer3 --sshlogin : -j 1 --progress --verbose --joblog parallel.log

I already had SSH key-based authentication set up so that I could connect to the three remote computers.

Then I spread the jobs over four computers with the following command:

cat originals.txt | parallel -J emacsconf \
                             --transferfile {} \
                             --return '{=$_ =~ s/\..*?$/--compressed32.webm/=}' \
                             --cleanup \
                             --basefile compress-video-low.sh \
                             bash compress-video-low.sh 32 {}

It copied each file over to the computer it was assigned to, processed the file, and then copied the file back.

It was also helpful to occasionally do echo 'killall -9 ffmpeg' | parallel -J emacsconf -j 1 --onall if I cancelled a run.

It still took a long time, but less than it would have if any one computer had to crunch through everything on its own.

This was much better than my previous way of doing things, which involved copying the files over, running ffmpeg commands, copying the files back, and getting somewhat confused about which directory I was in and which file I assigned where and what to do about incompletely-encoded files.

I sometimes ran into problems with incompletely-encoded files because I'd cancelled the FFmpeg process. Even though ffprobe said the files were long, they were missing a large chunk of video at the end. I added a compile-media-verify-video-frames function to compile-media.el so that I could get the last few seconds of frames, compare them against the duration, and report an error if there was a big gap.

Then I changed emacsconf-publish.el to use the new filenames, and I regenerated all the pages. For EmacsConf 2020, I used some Emacs Lisp to update the files. I'm not particularly fond of wrangling video files (lots of waiting, high chance of error), but I'm glad I got the computers to work together.

View or add comments

Toggle screen recording with AutoKey and vokoscreenNG

| geek

I want to be able to toggle recording with a keypress, but vokoscreen-ng didn't seem to have a shortcut for that. Autokey to the rescue! At first I tried to send <ctrl><shift><f10> directly, but that didn't work because I had to fake the keypresses. Here's the working script:

from subprocess import call
keyboard.press_key("<ctrl>")
keyboard.press_key("<shift>")
if not store.has_key('voko-running'):
    store.set_value('voko-running', 1)
    keyboard.fake_keypress('<f10>')
else:
    store.remove_value('voko-running')
    keyboard.fake_keypress('<f11>')
keyboard.release_key("<ctrl>")
keyboard.release_key("<shift>")

I've bound it to my Menu key. Let's see if that makes things easier.

View or add comments

2021-12-20 Emacs news

| emacs, emacs-news

Links from reddit.com/r/emacs, r/orgmode, r/spacemacs, r/planetemacs, Hacker News, planet.emacslife.com, YouTube, the Emacs NEWS file, Emacs Calendar, emacs-devel, and lemmy/c/emacs.

View or add comments

Why I Love Emacs - from Bob Oliver

| emacs, org

Sometimes I post updates from people who don't have their own blog. Here's one from Bob Oliver. - Sacha

This short article sets out why I, as an Emacs newbie, really, really love this software. But before I get into that I would like to explain my voyage (Note: absence of the 'journey' word) to Emacs.

Many moons ago, back in the late seventies / early eighties I was a Cobol programmer, a job I loved. As it is with life, circumstances change and I moved away from Data Processing, as we called it in olden days. This meant I had to get my programming fix using my Sinclair Spectrum, which I programmed using their version of BASIC. I learned how to build my own, very simple games, and spent many hours playing my games and programming more. Then the children came along, the Sinclair went into the loft (attic for non-UK readers) and I had little or no time for hobbies.

Years later, with family grown and flown the nest, the Raspberry Pi was released and revised my love of programming. I took to learning C and Python - though remain very much at the beginner stage. All very enjoyable. This sparked a notion that I might be able to build an app and enhance my future pension prospects. To this end I installed xCode on my MacBook and also tried VS-Code. Needless to say I have not achieved proficiency and have since removed those products from my MacBook.

I still wanted to enhance my knowledge of C, Python and Bash, and so was really pleased when the Raspberry Pi foundation released Raspberry O/S Desktop for Mac (apologies if this name is not technically correct). This enabled me to re-purpose an old MacBook (circa 2009 and no longer supported) as a Linux machine, which got me interesting in learning all things Linux. This led to me installing Emacs as my code editor. Through reading all things Emacs I discovered org-mode and now Emacs is my text editor of choice.

As probably most new users to Emacs, I found it a bit confusing at first, but did as recommended stuck with it, and I am really glad I did.

What do I use Emacs for?

A very good question. Short answer is code and text editor.

  1. Writing, compiling, testing and running C programs.
  2. Writing, testing and running Bash scripts.
  3. Writing, testing and running Python programs.
  4. Compiling my, not so, daily journal.
  5. Using org-mode as my word processor of choice.

The key reason for using org-mode for my journal, was portability and long term accessibility. I had used various electronic journals before, each with their own proprietary file standards, making me concerned that my journal would not be available to my children long after I have gone. Also as Linux, and hence org-mode, use plain text files I can edit with any text editor on any platform, so can be assured that I can move the files as and when I change computers. Also as plain text files, they are readily searchable, so I can recall memories easily.

Finding Emacs and org-mode is probably one of the best things I have done since I retired from full-time employment.

What next:

  1. Maintain my journal writing.
  2. Write up my poems in org-mode - I have several going back to my teenage years.
  3. Develop my writing skills and maybe write a novel.
  4. Learn how to send and recieve mail through Emacs - I have yet to find a guide that is not too technical / complicated for me.

SO MY MESSAGE IS JOIN THE EMACS AND ORG SOCIETY - YOU WON'T REGRET IT.

Bob Oliver Essex, England.

View or add comments

Thinking about Emacs community maintenance

| emacs

Building on Bastien Guerry's talk on maintainers at the recent EmacsConf, I thought I'd reflect a little bit on what I do to help out with the Emacs community because, as Bastien pointed out in his Q&A, that's also kind of maintenance. If I think about what I like to do and why, and what I don't like to do and why not, I'm more likely to keep enjoying things and not burn out. If I think about what I do, I might be able to find ways to do them more effectively or other things that complement them efficiently. And hey, maybe those notes will prompt other people to think about what they do, what they like and dislike, and how to grow.

I like helping with Emacs because it tickles my brain. I like playing around with Emacs and getting it to do interesting things, and I get a lot of reward for a little effort. I do this because it's intrinsically fun. Hacking on cool things with Emacs is interesting because I can come up with different things that a text editor really shouldn't be able to do, and yet it does. I also like coming across blog posts by other people who have written about their ideas, or maybe they're building on something that I've shared, or maybe they've come up with ideas of their own. It's great when other people share, as I get to learn from them too. And of course, you know, of course, comments and emails and all that stuff are also very nice.

There are a few things that aren't a particularly good fit for me or my life at the moment, so I try to do as few of them as possible.

  • Time-sensitive commitments: meetings, coordinating with other people, hosting video chats, scheduled livestreaming… It would be nice to help out, but I also need to hang out with the kiddo. I don't like the feeling of being pulled two different ways, so that's something I've been minimizing.
  • Since I don't have much computer time, I'm still a little too impatient to listen to or watch videos, and I like my quiet time too much to listen to them when I'm washing the dishes or tidying up. I don't have a lot of discretionary time, so I'd prefer to speed-read through captions.
  • Talking is a little tough. After a day with a little kid, sometimes I don't particularly want to talk to anyone. I'm experimenting with recording myself thinking out loud so that I can get ideas down quickly, but I'm not quite at the point of doing livestreams yet.
  • Keeping things in my head is hard, too, like if I'm working on a big task that's hard to break down into small chunks, or something that requires a lot of these small chunks built up over a period of of days. My discretionary time is really one- or two-hour chunks (or less) and pretty fragmented fragmented–one day here, another day there–because I've got other things that I need to do in the meantime. So I tend to focus on short things instead.
  • I'm also not really keen on things that build up more technical debt. There's a whole list of things that I will get to do some day when I can actually sit down and focus on things. In the meantime, I just want to get through this part. =)
  • I'm kinda iffy on e-mail. I tend to respond slowly, and I usually redirect people to other places like #emacs or Reddit if they're asking for help. Don't have much time at night, and I don't want to spend it going through my inbox.
  • Emacs geeking becomes less fun if things aren't smooth at home: grumpy kiddo, clutter building up, too many oops moments.

Anyway, so the core of what I do:

  • Emacs News: I put that together every week. I browse Reddit posts as a mental break from hanging out with the kiddo. I pick up other sources, too. People email me interesting things to include. I've got a few shortcuts for filing things. Putting it all together takes about 30-60 minutes, especially if I get carried away incorporating cool stuff into my config. I do it because it's a lot of fun to see all the cool things that people are working on, and it takes advantage of speed-reading. Plus I can do most of the information-gathering in the scattered moments of my day, even if I don't have a chunk of focused time to come up with and write about an interesting Emacs hack. And it seems to be a great way to link the community up, because there's just so much stuff that's going on.
  • Code: I share my config and I add to it once in a while. Lately I've been breaking things out into smaller modules that perhaps someday could turn into packages. For the most part, I stick things into my config. Some of those functions turn into blog posts as well.
  • Blog posts: They're a great way to remember, share, and get ideas and comments from other people. It's particularly fun when other people build on those ideas and then I get to use their even awesomer version.
  • EmacsConf: This feels like it takes a surprising amount of time. I like working behind the scenes. I like doing the schedule because it tickles my brain when I can put the talks in some kind of flow. I like doing captions because I can work with text. I do a lot of the publishing on the wiki, and I've written a lot of automation for it. It's a good part of my evil plan, because it gets stuff out of people's heads and configurations and into a form that other people can look at and maybe talk to them about it. It's handy for connecting people.
  • Calendar: This is basically an iCal aggregator that gets updates from various meetup groups and includes some manually-created entries. It goes into a bunch of places, like Emacs News, #emacs, EmacsWiki:Usergroups, and Twitter. I think that's a nice little way to help people find virtual events, and then people don't feel so isolated. I find that a lot of people are the only Emacs users they know in real life, so it's fantastic that there are a lot of virtual meetups these days. Then they can find other people who can share ideas. I think that one of the best ways to learn Emacs is by looking over someone's shoulder, because then you can you can see things that the other person might forget to point out to you, and you can ask questions about those. Meetups are great for that sort of thing, and that's why I do the calendar as well.

I don't have as much time for things like sketches or videos at the moment, but I like to do them once in a while. Sketches make it easy to map things out, untangle my thoughts, or plan ideas. Some of my beginner maps are still floating around. Videos are handy for showing interesting things. Sometimes people ask questions about things that aren't actually the focus of the video, which is great. Hanging out in #emacs is nice because I learn from other people's questions and answers.

It would be fantastic if someone could do emacs-devel summaries which I could just link to in Emacs News. The traffic is a bit too heavy on that list for me to get a sense of how to summarize it, and I have a hard time keeping track of things. I'd also love it if someone else did video encoding and other detail-oriented work, so if someone wants to take that on for EmacsConf, feel free. Someday it would be nice to have enough free time to go to virtual meetups or do Emacs Chats again, and it would be nice to see more people organizing those as well. And more blog posts, too! I'd also love to have more playtime to experiment with interesting packages and features, too. Anyway, someday!

In the meantime, I've been experimenting with tweaking my workflow. Lately I've been actually plugging into an external monitor instead of trying to code on my laptop. I've also got to work on the sleep/discretionary time trade-off. If I stay up too much, the next day gets a bit rough, and rough is hard with a kiddo. It'll be interesting to smoothen my workflow for posts, especially if I can get the hang of including images and screencasts, because then I can post more things quickly, which means I can squeeze them into the time that I have.

So those are basically the things I do for Emacs community maintenance. I'm kinda in a holding pattern while I'm still dealing with all the things that come with life with a small child, but kids grow faster than one might expect. I'll be able to get back to longer posts someday, but right now, this is good. Anyway, that's what I do because it's fun.

Hmm… Now that I think about it, the ideal things for me to work on:

  • are self-directed and low-risk, so that when I'm not working on them, I can fully focus on playing with A- instead of getting pulled back and forth by coordination or dealing with mistakes
  • can be broken down into small tasks (maybe 30min to an hour) and written about, or can fit into 5 minutes here and 5 minutes there, or can build on things I can do during non-screentime
  • aren't error-prone, since I'll be doing them late at night and possibly a bit sleepily (I tried early morning, but A- wakes up a bit unpredictably)
  • are text-based (easier to edit, update, share, and search), so I don't need to fiddle with video/audio for now
  • can result in compounding benefits, like automating a process or connecting people

So I might swap some of the things that don't fit that profile for things that do. I think that mostly means working on smaller chunks and writing more (possibly using autogenerated transcripts to split writing into recording and editing), and handing over some of the tasks I don't like so much so that other people can pick those tasks up if they want.

It might also be worth channeling people's goodwill into better things, too. For example, if people happen to want to thank me for Emacs News or other things, perhaps I can convince people to write a blog post and/or Reddit post sharing something Emacs-related (interesting thing you learned recently? a package that more people should know about? your story of how you got into it and what you've learned along the way?), since that can go out into the wider world and expand the conversation.

There's plenty of room to make things even more fun and more effective, so let's see. =)

View or add comments

Adding little nudges to help on the EmacsConf wiki

| emacs, emacsconf

A number of people helped capture the talks for EmacsConf 2021, which was fantastic because we were able to stream all of the first day's talks with open captions and most of the second day's talks too. Right now, in fact, there are only two talks left that haven't been captioned. After the conference, a couple of other people volunteered to help out as well. Whee!

I want to figure out a good way to help people work on the things that they're interested in without necessarily burdening them with too much work, too little work, too much coordination, not enough coordination. Before the conference, one of the perks we had offered was that captioners got early access to the videos. I had a password-protected directory on a web server and an index that I made using Emacs Lisp to display the the talks that still need to be captioned. People e-mailed me to call dibs on the talk they wanted to caption, and that was how we avoided duplicating work. Now that all the videos are public, of course, people can just go to the regular wiki.

The other thing to think about is that in addition to captioning the two remaining talks (not essential, but it would be nice), there are also different levels of things that we can do. It would be nice to have chapter markers for some of the longer Q&A sessions. It would be fantastic to cross reference those with the questions and answers so that so that people can jump to the section they're interested in. It'd be incredible if somebody actually wrote down the answers. And it'd be even more awesome if people actually captioned the Q&A sessions as well, which were in many cases much longer than the actual sessions. So this is a fair bit of work, but people can probably pick a level that matches their interest and time available.

I'm not entirely sure how to coordinate this especially since I've got limited computer time. So my goal is to have something where volunteers can basically just wander around looking for talks that they're interested in and see ways to help out, or see a list of things that could use some work. So for example, while they're browsing the maintainers talk, they might say, "Oh, this one needs some chapter markers. I want to help with that. How do I do that? How do I get started?" And then they go down that path. On the other hand, you might have somebody sitting down saying, "I've got an hour and I want to go help out. What can I do?"

I don't want to keep data in many different places. I wonder if I can use the wiki for a lot of this coordination. Now that the videos are public, I've started tagging the pages that need extra help, like long Q&A session that need chapter markers.

With a little bit more work, I think people will be able to follow the instructions from there, especially if they've done this kind of captioning before, or email us to ask for help and then we can get them started.

I also thought about using Etherpad to do that kind of coordination where people would put their name next to a thing to reserve it, but then that's one more step. I don't know. At the moment, editing the wiki is a bit of an involved process. Worst-case scenario (best-case, actually, if we have lots of people wanting to help? =) ), people can call dibs by emailing us at and one of us organizers will add a little note there in the volunteer attribute. It's probably a good start, so we'll see where we can take it.

View or add comments

Putting off laptop upgrade again

| geek

I'm still using my Lenovo X220 tablet PC from 2011. Actually, technically, I'm using W-'s, since he got the same model and he's not using his I was thinking about whether I want to upgrade my laptop. It looks like GNU Parallel will let me do my video encoding across multiple nodes, so I don't necessarily need to upgrade just to be able to handle the once-a-year ffmpeg encoding of all the EmacsConf videos. Recording videos and live streaming is mostly bottlenecked by me not wanting to talk much at the end of the day, but I can experiment with that even with my current setup. Plugging in modern external monitors, well, there's a VGA monitor downstairs that I haven't been using. So I'm going to try that first. Better resolution would be nice, but we're going to start with the external monitor, I think. If I find myself actually be able to do more focused work, then it might make sense to get something that can do 1920x.

So yeah, I think I can put off an upgrade for another year. I did also look into the possibilities of just upgrading the RAM on my laptop, since 16GB will run me maybe ~$200. Not bad. I should find out how often I actually swap. And if I ever run out of space as well (sometimes I need to reshuffle how my my files are organized), apparently, they have 4TB SSDs these days. Still, it's not a big deal.

I think I'm just gonna keep waiting. I'll try using an external monitor at night, so maybe I'll leave my laptop down and just patch it into that VGA monitor once in a while. I'm going to try to do more thinking, planning, and writing.

I'll check out this again once the kiddo settles into in-person school, whenever that will be. Then maybe I'll have more focused time to do some programming and other cool stuff.

View or add comments