The Esoteric Magnificence of Org

Published May 20, 2024 · 2959 words · 14 minute read

Of all the productivity and thinking tools I’ve come across, all shake and tremble beneath the brilliant and towering presence of org-mode in all of my note-taking, organization, scheming, and tinkering. Over the past four months, my workflow has entirely surrendered to the insanely flexible power of the EMACS text editor and community-driven libraries.

…that is exceedingly high praise for a text editor. I promise by the end of this article I’ll at least have you curious about this profoundly excellent system for managing your thoughts, work, and life.


Org-roam Notes Network of Peter Prevos
Org-roam Notes Network of Peter Prevos


Org is a highly flexible structured plain text file format, composed of a few simple, yet versatile, structures — constructed to be both simple enough for the novice and powerful enough for the expert. (src )

Nearly every Org user has a story to tell about how Org enables and empowers them — some have found it so useful that they have written (scientific) papers about the value of Org for conducting reproducible research. (src )

What is Emacs? What is Org-Mode?

Emacs is a text editor1 and Org-Mode is an operating mode2 for the editor.

Emacs is special because it is built around an interpreter for a programming language called Emacs Lisp , a functional programming language that enables developers to write very powerful and flexible programs. Because the editor is a Lisp program, it is very easy to update the program while it is running, allowing a user like you to change the editor as it is running.

This venerable text editor is an extremely long-lived3 community project. EMACS will certainly outlast all other text editors. It first emerged in 19764, and its record of published versions (starting with 13) goes back to 19855. When I was born, the current release was 20, and now we’re up to 29!

Org-Mode is automatically started when you begin editing an .org file in EMACS. Org files are a lot like Markdown files - everything is in plaintext, and the files can be read easily by people with any editor. In other words: the data is completely within your control, and can be viewed and modified with or without a copy of the org program. By contrast, files in Word or OneNote are tough to decipher if you try to open them without the programs that created them.

Together, Emacs with Org-Mode is powerful enough, but as stated earlier, it is extremely easy to customize the Emacs editor to your taste on the fly.

If you’ve read or written Racket or Scheme before, Emacs Lisp (elisp) will be very familiar to you. The manual may help.

The syntax for lisps is simple:

;; Comments start with semicolons

;; A lisp form
(function-name argument-1 argument-2 argument-3)

;; Placing a ' at the start of a form makes it a list
;;  and informs the compiler "evaluate this form as data"
'(1 2 3 4 5)

;; Refer to functions outside of calls using #'
(apply #'function-name 1 2 3)

…apart from macros, that’s all the syntax there is. Nice, right?

With this programming language, you can build modes or extensions for the Emacs editor. Even faster libraries from any compiled language are easy to call. The ease of working with elisp allows the editor and the libraries you install to conform to your habits and taste, making the living systems created with Emacs very personal.

Execute Code Blocks in Any Language

Literate programming was first conceived by Donald Knuth in 1984 and involves the interleaving of a program with its documentation, enabling more thorough explanation to be presented alongside source code. This is most popular today in the form of Jupyter notebooks for enabling data science and machine learning in Python.

Allow me to shout this ultra cool feature from the rooftops:

Org enables you to execute any6 language inline!

This means you can mix all of your favourite languages in a single file and have the output for each execution printed directly and effortlessly back into your org file. In the editor, an inline code block looks like this - you only need to hit C-c C-c to execute it:

Car grabs the first element of a list.

#+begin_src lisp
  (car '(pork beef chicken shrimp))
#+end_src

#+RESULTS:
: PORK

Here’s a second example with some inline Clojure:

*get* allows you to grab keys, and can return nil or a default:

#+begin_src clojure
(get {:x 1 :y 2} :y)   ;; => 2
(get {:x 1 :y 2} :z)   ;; => nil
(get {:x 1 :y 2} :z 3) ;; => 3
#+end_src
*get-in* allows you to dig into nested maps:

#+begin_src clojure
(get-in 
  {:head 1 :chest {:ribs 10 :cavity {:heart "pumpin'" :lungs 2}}} 
  [:chest :cavity :heart])
#+end_src

#+RESULTS:
: pumpin'

My work-in-progress Clojure manual is actually an org-mode file. I’m not just writing code on the side and pasting it into the file - the file itself has a first-class connection to the REPL.

By using org, there is no difference between programming and writing the aforementioned webpage for my own future reference. With the ~tangle~ command, you can extract selected code blocks into an external script, enabling the writing of complex code within a system in a literate manner.

Aside: Some people even write their EMACS settings in an ORG file .

In-File Tables & Table Formulas

Not only does org make it easy to create and navigate tables in plaintext, but it provides an easy way to write formulas that can be applied to these plaintext tables!

Here’s an example of a simple table I have:

| Date             | Service                             |   Cost |
|------------------+-------------------------------------+--------|
| [2022-10-05 Wed] | Oil Change                          |      0 |
| [2023-05-23 Tue] | Oil Change & Recall                 |      0 |
| [2023-09-25 Mon] | Oil Change, Trans. Fluid, Brakes    | 674.30 |
| [2023-11-29 Wed] | Self: Antifreeze fluid, air filter  |  68.72 |
| [2023-12-02 Sat] | Self: Rust Inhibitor, Chip Paint    |  70.04 |
| [2024-01-03 Wed] | Fix chip in windshield              |  15.73 |
| [2024-02-14 Wed] | Oil Change & Cooling system service | 281.96 |
| [2024-04-15 Mon] | Travis - Inspect Pothole Damage     |  50.00 |
| [2024-04-26 Fri] | Replace Stabilizer Links            | 376.08 |

…what if I want a running total of how much I’ve spent on my car?

Using Emacs Calc syntax , I can do this by adding a column, typing C-c = to add a formula for the column, and using the formula $4=vsum(@I$3..@$3).

| Date             | Service                             |   Cost | Total [c] |
|------------------+-------------------------------------+--------+-----------|
| [2022-10-05 Wed] | Oil Change                          |      0 |      0.00 |
| [2023-05-23 Tue] | Oil Change & Recall                 |      0 |      0.00 |
| [2023-09-25 Mon] | Oil Change, Trans. Fluid, Brakes    | 674.30 |    674.30 |
| [2023-11-29 Wed] | Self: Antifreeze fluid, air filter  |  68.72 |    743.02 |
| [2023-12-02 Sat] | Self: Rust Inhibitor, Chip Paint    |  70.04 |    813.06 |
| [2024-01-03 Wed] | Fix chip in windshield              |  15.73 |    828.79 |
| [2024-02-14 Wed] | Oil Change & Cooling system service | 281.96 |   1110.75 |
| [2024-04-15 Mon] | Travis - Inspect Pothole Damage     |  50.00 |   1160.75 |
| [2024-04-26 Fri] | Replace Stabilizer Links            | 376.08 |   1536.83 |
#+TBLFM: $4=vsum(@I$3..@$3);%.2f

Calc formulas have a similar function to Excel formulas. For more complex operations, Emacs has two more tricks up its sleeve:

  1. You can call elisp functions in these formulas7.
  2. You can use the data in code blocks for advanced processing.

Use Tables as Data in Code Blocks

If you wanted to take the latest (lowest) number in a particular column within your table and save that value for further use in other code blocks, here’s how you wire a table into your code block as a variable:

  1. Add a name property to the table #+NAME: abc-table
  2. Add a var property to the code block :var data=abc-table
Let's say we have a simple table of financial data:

#+NAME: big-loan
| Date             | Balance $CAD |
|------------------+--------------+
| [2023-01-30 Mon] | 16,128.39    |
| [2023-09-17 Sun] | 14,479.61    |
| [2023-12-23 Sat] | 12,332.19    |
| [2024-03-02 Sat] | 11,379.16    |
| [2024-03-03 Sun] | 9,120.16     |
| [2024-04-12 Fri] | 8,784.89     |
| [2024-04-27 Sat] | 7,808.79     |

We can grab the first row of the table:

#+begin_src elisp :var data=big-loan
(car data)
#+end_src

#+RESULTS:
| [2023-01-30 Mon] | 16,128.39 |

We can print the last updated balance as a dollar amount:

#+begin_src elisp :var data=big-loan :exports both
(setq big-loan-cad (to-num (col-last data 1)))
(render-human-dollars big-loan-cad)
#+end_src

#+RESULTS:
: $7,808.79

The functions called here are all custom and loaded when Emacs is started. Here’s the function for picking out the last entry in a column, though you’ll need to learn a bit more about lisp to understand it:

(defun col-last (data col-index)
  (nth col-index (caar (list (last data)))))

Again, the beautiful thing about Emacs is its customizability - I wrote this short function to re-run all of the table calculations, then run all the code blocks in the current file.

(defun rcf/recompute ()
  (interactive)
  (org-table-recalculate-buffer-tables)
  (org-babel-execute-buffer))

(global-set-key (kbd "C-c o r") 'rcf/recompute)

All I need to do now is hit C-c o r and all my computations are re-run.

Build an Idea Graph with Org-Roam

When taking notes, have you ever been frustrated when realizing that you have information on one topic scattered into several different notes or places? Org-roam can fix that - it’s like building a second brain8.

  • org-roam enables note-taking with the Zettelkasten methodology.
  • org-roam-ui allows you to beautifully visualize your nodes
  • org-roam-protocol allows you to use browser extensions and external tools to build your node web

An example from Alex Kehayias’s notes
An example from Alex Kehayias’s notes

Building out this graph is as easy:

  • Type C-c r f to find an existing node or create a new one
  • Type C-c r i to insert a reference to a new or existing node

This is accomplished with some simple keybindings:

(global-set-key (kbd "C-c r f") #'org-roam-node-find)
(global-set-key (kbd "C-c r i") #'org-roam-node-insert)

Be mindful when building your knowledge graph. It is easy to copy-paste in bucketfuls of data, but the Zettelkasten method when applied correctly encourages brevity.

Agenda and ToDo Tracker

Holding shift on a headline allows you to cycle between states, to TODO or DONE. Typing C-c a t allows you to see ToDos from all your Org files, and indicates the file the todo is in and any tags or priorities set. You can set a custom sequence on a per-file basis:

#+SEQ_TODO: TODO(t) FOLLOW-UP(f) RECURRING(r) IN_PROGRESS(p) STALE(s) | DONE(d) CANCELLED(c)

You can use C-c C-s to schedule a time for a task, or C-c C-d to set a deadline with a day and time. All these will show up in your agenda.

Week Agenda - Bill de Hora - How I Use Org Mode
Week Agenda - Bill de Hora - How I Use Org Mode

With these tools alone, Org becomes an extremely powerful way to organize all of your ToDos.

Even if you add a date under a heading with C-c ., it’ll show up in the agenda so you can keep track of when notes were taken.

Org-agenda allows you to see at a glance all of your organized TODOs in a linear calendar, in a list, or filtered by a tag applied to a headline with C-c C-q. Checking all of your actions related to a project, IT ticket, or week is a breeze.

Agenda Commands - Caleb Rogers - How I Stay Organized using Emacs
Agenda Commands - Caleb Rogers - How I Stay Organized using Emacs

Time Tracking

In addition to pre-emptively organizing your time, you can also keep track of how long you spend on any given task and get a clear overview of where your time is going.

If you are taking notes beneath a header, tapping C-c C-x C-i will clock in and C-c C-x C-o will clock out.

If you expand the properties under the headline that’ll look like this:

** FINISHED Rich Dad, Poor Dad
:LOGBOOK:
CLOCK: [2023-12-15 Fri 19:31]--[2023-12-15 Fri 20:28] =>  0:57
CLOCK: [2023-12-15 Fri 13:21]--[2023-12-15 Fri 13:59] =>  0:38
:END:

A clocking table can be added anywhere in the file, with configurable scope and depth (how many headers will be shown before times are combined).

#+BEGIN: clocktable :scope file :maxlevel 3
#+CAPTION: Clock summary at [2024-01-26 Fri 18:02]
| Headline                           | Time |      |
|------------------------------------+------+------|
| Total time                         | 2:26 |      |
|------------------------------------+------+------|
| Money - Finance, Saving, Investing | 2:26 |      |
| \_  Rich Dad, Poor Dad             |      | 1:35 |
| \_  Multiple Streams of Income     |      | 0:51 |
#+END:

You can jump to the current clock task/header with C-c C-x C-j, and recompute the table or time interval with C-c C-c while the cursor is placed over top of these respective items.

Searchability - GREPing your Stuff

Would you like to hit a single key to do a full-text search of all your org files? If you’ve installed the projectile package, you can add this bit of elisp to your config:

;; projectile is another amazing package from the
;; creator of CIDER. It's got lots of commands
;; for searching and managing files in a project.
;; https://projectile.mx/

(setup (:package projectile)
  (projectile-mode +1)
  (:bind "s-p" projectile-command-map
         "C-c p" projectile-command-map)

  (global-set-key [f9] 'projectile-grep))

Now, hitting f9 will do a full-text search on all your Org files, and anything else in the directory. You can exclude files and paths in a .projectile configuration file at the root of your projects folder.

LaTeX Export - Or Any Format, Really

You can write LaTeX inline in org files. Mathematics formulas included.

Output can be customized easily with some headers:

#+LATEX_CLASS_OPTIONS: [letterpaper,10pt]
#+LATEX_HEADER: \usepackage[margin=1.0in]{geometry}

…far deeper export customization is available, but I haven’t leveraged it yet.

Typing “org export” into M-x for me yields 196 options with my current configuration. Many of these are the same file format via a different technology. For example, there are 15 different methods of exporting to a PDF.

On a Mac, you can export to PDF with M-x org-latex-export-to-pdf - though LaTeX export won’t work on OSX without:

╭─[email protected] /Library/TeX/texbin
╰─➤  sudo tlmgr install wrapfig marvosym wasysym capt-of

Enjoy your beautiful PDF export!

Storing Knowledge as Plaintext

This is a more philosphical aspect of computing in this manner:

Plaintext is forever.

By storing things in plaintext, you’ll never lose access, you’ll always be able to rebuild whatever functionality was lost, and most importantly, you have complete control over your tools and the fredom to choose them.

You can read and write plaintext with an innumerable number of programs.

You can easily reprocess plaintext into a variety of other formats and websites with ease.

Locking your wisdom and data in a proprietary format makes it difficult to synchronize and share, and makes you dependant on large companies that positively hate your guts and see you only as ad dollars.

Do you trust those people with your notes?

Extending Org-Mode and Emacs

A good portion of the things I’ve described above have been combined in a unique way or altered to my taste. The best part about a programmable text editor is that it’s extremely easy to hack on and change to fit your working style.

While it’s not for the faint of heart, Emacs is a journey worth taking, something I wish I pursued far sooner in my life as it certainly would have made a positive impact on my University grades.

Here are some more inspirational sites to start you on your Emacs journey:

  1. orgmode.org - simple Org tutorial
  2. github.com/emacs-tw/ - awesome-emacs#table-of-contents
  3. orgmode.org - org-mode hacks
  4. systemcrafters.net - basics-of-emacs-configuration
  5. wilfred.me.uk .emacs.d/init.el
  6. github.com/caisah/ #a-list-of-people-with-nice-emacs-config-files
  7. config.phundrak.com - org.el
  8. sqrtminusone.xyz - emacs.el
  9. www.patrickdelliott.com - emacs.d/

Summary

Org-mode along with some additional packages enables you to:

  1. Build a body of personal knowledge in plaintext
  2. Organize that body with files, headlines, and Org-roam
  3. Instantly search all of this with grep
  4. Include data in tables with calculated columns
  5. Manipulate those data tables in any programming language
  6. Manage all of this in varying states of completion
  7. Add dates, times, priorities, and time estimates
  8. Easy access to the source code to tweak things to your taste
  9. All of this is completely under your control

Conclusion

This article is more a case of tell than show.

What I hope you take away from this is that you have freedom in your ability to manipulate and manage your own data. There are brilliant tools outside of the walled gardens to manage and visualize your time and thoughts.

If you have the wherewithal, I’d like you to take a moment and try Emacs on for size. The zen of having a single extensible place to read, write, manage, and think is truly glorious.


This article was researched and written entirely without the use of AI.


  1. Text Editor: Calling Emacs a text editor is akin to calling a computer a calculator. It is far closer to a philosophy or an application framework than just an editor. ↩︎

  2. Operating Mode: When opening a file, Emacs can apply a major mode and any number of additional minor modes that add functionality like spell checking, bracket matching, highlighting, etc. ↩︎

  3. Bozhidar Batzov, Forever Emacs , 2022 ↩︎

  4. Wikipedia: GNU Emacs  ↩︎

  5. GNU Project: Emacs History  ↩︎

  6. Any Language: Caveat: Any language with an interpreter or executable that can run on your system. It’s not magic, but it is unbelievably flexible. ↩︎

  7. Org Mode Docs: Org spreadsheet LISP formulas  ↩︎

  8. Org-roam Manual: Introduction  ↩︎

Comments