How do you buffer a Gemini connection?

Omar Polo op at omarpolo.com
Mon Mar 29 09:13:07 BST 2021


almaember <almaember at disroot.org> writes:

> Hello,
>
> I'm planning on creating my own Gemini client (with C99 and OpenSSL, if
> you want to know). However, I have a problem with how I should be
> implementing the downloading part.
>
> So, how should I be buffering a Gemini connection? Since there is no
> size indicator, I can't be sure everything even fits into memory, or
> how much time it's going to take to download, or if it's even going to
> end at all.
>
> Another problem is long-polling. Through my exploration of Geminispace,
> I found a few capsules that used a hack to push live updates to the
> client. Specifically, these servers didn't close the connection, but
> instead kept it open and just sent new data when something happened.
>
> Since the spec says nothing (or I'm blind) about buffering, so I want
> to ask others about what I should be doing, while still being able to
> actually parse the output.
>
> My ideas as of right now:
>
>  - Have a big buffer and store everything in it until the connection is
>    closed. The connection would be terminated if:
>     - the connection closes
>     - the server didn't send anything for X seconds
>  - Buffer by line and don't close the connection unless the user
>    terminates it (by pressing the stop key or by loading another
>    capsule). Compatible with long polling.
>
> Thanks in advance for any help!
>
>
> ~almaember

I'm working on a client and doing exactly as Ecmel Berk Canlıer outlined
in his mail.  I'm fetching the page one chunk at the time (it allows me
to use a fixed buffer and plays nice with asynchronous I/O) then get
sent to a proper renderer (atm only text/gemini and one generic text/*
handler).  There it gets parsed and split line by line.

The only part where I have to actually keep a dynamic buffer is when
splitting in lines, because a line can (and often is) splitted across
chunks, and possibly also bigger than one chunk.

I'm using LibreSSL (plus some other OpenBSD goodies), but if it saves
you some time here is:

* the protocol code: https://github.com/omar-polo/telescope/blob/main/gemini.c
* generic code shared by the parsers: https://github.com/omar-polo/telescope/blob/main/parser.c
* text/plain "parser": https://github.com/omar-polo/telescope/blob/main/textplain.c

Warning: the client is not yet 100% functional, but I'm quite found of
this design.  One additional benefit is that you can also render the
page on the go, and that enables also to stream text/* stuff.

Cheers,

Omar Polo


More information about the Gemini mailing list