Writing my own calendar syncing solution

Introduction

I am using org-mode to organize all my tasks, personal and work-related, and it works great for me. But lately, I have realized that it would be even more useful, if I could also access it on the fly on my phone and synchronize it more easily across devices. The problem is of course that org-mode only lives within Emacs. There are solutions like Orgzly ¹ that I have used in the past and they work well, but I would like to see my to-do-list within the tasks app that I am already using. And this should be possible, as I am using CalDav compatible clients everywhere, managed via my own Nextcloud instance.

Now, there is actually an Emacs package (because of course there is) that allows to sync org files with CalDav calendars. Unfortunately, it doesn’t really work for me. I hit a few problems:

1. It is very slow. No idea why, everywhere else the communication with the server is fast enough, but Emacs grinds to a halt for several minutes syncing one event after the other, taking multiple seconds for each. Sometimes I am wondering, if the process has crashed - that’s how slow it is. This could totally be on me for not configuring something correctly or be due to my Nextcloud instance, but either way, it limits the usefulness immensely. ²
2. It is opiniated in a way that is not compatible with my workflow. Headlines without a TODO tag and a timestamp are considered events. Fair enough, but I do not use headlines without TODO tags, everything is a todo item, a meeting or an appointment as well. I do that to enable time tracking there as well. Every headline with a TODO and a timestamp gets turned into a task. That could work for me, but tasks are shown as small entries in my calendar, not as time blocks. Maybe that is a limitation of the Nextcloud calendar app, but the system just does not work for me like that. Also, the headline needs a deadline to be shown. I just schedule things, I do not use deadlines for that.
3. I am not sure, if I just have not set it up properly or if there are bugs, but half of the time I get sync errors with duplicate IDs or items not recognized. Deleting everything and starting over does not solve it. Even though there are no duplicates around anymore. I guess something is being cached somewhere, but I don’t know how to delete that. Long story short, it works about 1 out of 5 times.

After multiple hours of unsuccessfully trying to sync a blank calendar/todo list with some dummy tasks or events, I finally gave up. It is more work than what it is worth, when it worked it seemed very unstable, and I decided there has to be another way. Luckily, CalDav is not a new protocol and there are libraries around that deal with all the internals. And org files are just text, so I can write my own syncing solution. So I set out to do just that. This post is a little bit about the journey and some code that was created along the way. Enjoy!

The setup

So, in theory I would like to do this in Elisp, because then it could be integrated directly into Emacs and everything would be nice. However…I simply lack the skills for it. I never really warmed up to (E)Lisp, although I tried to learn it a few times, and am still intending to do so. Maybe this would actually be a good starting project, but I would prefer to have a quick solution first. My goal is actually to rewrite this tool in Elisp some day just for the challenge. In the meantime, I will do it in Python, because I know Python well and am very productive with it. Speed is not an issue, because the bottleneck will always be the server in this scenario. ³

I found a pretty mature solution to handle CalDav from within Python, caldav. It allows to communicate with CalDav servers, create/edit/delete calendars, events, and tasks. Its API is sometimes a bit strange, but it does its job well. org files can be parsed with the package org-parse, and that’s all that I need except some small helper packages like pytz.

So, let’s have a look. Getting a calendar is pretty easy:

1
2
3
4
5
6
7
8

import caldav

with caldav.DAVClient(
        url=CALDAV_URL,
        username=CALDAV_USERNAME,
        password=CALDAV_PASSWORD,
) as client:
        calendar = client.calendar(url=CALDAV_URL)

And then we can get all events or specific ones by their unique id:

1
2

all_events = calendar.events()
specific_event = calendar.event_by_uid('xxxx-xxx-xx-xxxx-xxxx')

Creating an event is also pretty easy:

1
2
3
4
5

calendar.save_event(
    dtstart=dt.datetime(2024, 5, 12, 12, 30),
    dtend=dt.datetime(2024, 5, 12, 12, 45),
    summary='Call somebody',
)

There is also a free-style description field that can hold any additional information, which can be useful to sync the body of org nodes or things like the LOGBOOK for time tracking. Right now I am just throwing the whole body of the org node into this field, i.e., everything after the properties drawer. There is surely room for some improvement in the future.

One thing that turned out to be harder than expected was updating events. To do so, we have to use some internal object that holds the ical data directly:

1
2

ical = event.icalendar_component
ical['SUMMARY'] = 'Call somebody important'

For timestamps we have to use a sub-property:

1

ical['DTSTART'].dt = dt.datetime(2024, 6, 12, 13, 0)

But in the case that the event does not have a DTEND property, e.g. because it was left empty at creation, this will crash. I resorted to always setting the end time to the start time, if not specified otherwise. A bit hacky, but works for now.

Accessing properties from an existing event can be done in the same way via the dictionary (and the sub-property dt in the case of timestamps). Any changes to this dictionary are only synced back to the server, once event.save() is called.

Timezones

Org timestamps do not contain timezone information, which makes sense, but a timezone-naive timestamp is treated as UTC by Caldav (at least in the calendar in my Nextcloud implementation). So here is a little code snippet to turn an org-mode timestamp like <2024-05-03 Tue 12:30> into a properly timezones datetime object, which can then be uploaded to the CalDav server:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

import datetime as dt
import pytz
import orgparse

def get_datetime_from_org(
    org_timestamp: str, timezone: str = 'Europe/Berlin'
) -> dt.datetime | None:
    if org_timestamp.startswith('[') or org_timestamp.startswith('<'):
        org_timestamp = org_timestamp[1:-1]

    dt_object = orgparse.date.OrgDate.from_str(org_timestamp).start

    return pytz.timezone(timezone).localize(dt_object)

Working with explicit timezones is good practice anyway, so no complaints here.

The sync logic

The software takes one or more org files, scans them for headlines, picks out all that are scheduled and not before some cutoff date, and compares them to the events on a specified cloud calendar. If there is no event with the same UID yet, then a new event is created. If there is one, then the fields of the remote event are overwritten with the values from org. In case of conflicts, the event is skipped. I had that happen, when I deleted an event manually on the server. In this case, the id is marked as “from a deleted event” and cannot be reused. Setting a new id in org solves that easily and the error message from CalDav is very clear about that. ⁴

For now, I am only syncing into one direction, org -> CalDav. That works well with my workflow, but I have to admit after using it for some days that being able to drag events around in the calendar web app is much more comfortable than rescheduling things in org-mode. Also, the visual aspect helps a lot to have a good overview and spot open time slots. The org agenda does a good job for a general overview, but cannot compete in my opinion. So the next big update of the tool would be to allow syncing changes back to the original org files, but then I have to decide how to handle conflicts and such. Right now I am simply overwriting whatever happens in the calendar.

Another nice addition would be to sync todos as actual tasks. So far I only turn org headings that are scheduled into calendar events. That is good enough for now, but I think turning headings with a deadline into tasks would be useful as well. Then I can decide, how I want to have my org entries synced to the cloud, as simple events in the calendar or as actionable items that I can manually check off. In that case the syncing from CalDav to org would be essential of course, because the DONE status must be reflected back to the org files.

One more thing would be a done calendar, meaning a separate calendar with all the events that are done. The logic would move any heading with a DONE keyword (so also stuff like CANCELLED or DELEGATED) to this separate calendar. In the web view I can then give that calendar a different color and then I can instantly see where I am over the course of the day, what stuff was left behind etc. This is not super important, but would be a nice little upgrade.

Putting it all together

The code is very much tailored towards my needs with little room for customization. If somebody wants to use it for their own tasks, it’s highly likely that some changes are needed. But it can definitely serve as a starting point or for learning how to deal with CalDav.

You can find all code on GitHub.

Notes