Instead of using Nix to build whatsxmpp, we switch back to using Docker in order
to make use of daewok's SBCL static executables fork
(ref: https://www.timmons.dev/posts/static-executables-with-sbcl-v2.html).
The Dockerfile, adapted from the sources of his fork, pulls his fork, builds
SBCL with sb-prelink-linkage-table, and then does the necessary stuff to make
a fully static whatsxmpp binary.
Importantly, the new binary uses SB-EXT:RESTRICT-COMPILER-POLICY to set 'safety
to 3 in all packages, to hopefully mitigate the heap corruption issues some
users are seeing. (The fact that Nix couldn't do this was a key reason for the
move away.)
We remove the old Nix files to avoid cruft / confusion.
Roster Item Exchange messages now add contacts to roster groups with names based
off the group chats you share with those contacts, to make organizing your
roster less painful.
This is both more informative than "whatscl on ...", and doesn't run the risk of
having an overly long description (by excluding the lisp version), which can
result in disconnections.
Sending lots of available presence is a battery drain, and also might
even get the WA app killed on iOS. (It also screws with your Last Seen,
which is probably not what you want.)
I somewhat forget why the code wanted to send so many presences in the
first place, but, anyway, to mitigate the issues above, let's disable
that for now.
- WARNING to operators: You MUST run fixup_2.sql in order to use this and newer
versions of the bridge.
- Archiving and full chat history fetches are now a configuration setting (and
default to off).
- Users now have to enable archiving manually by talking to the admin user, and
are warned about the potential privacy implications.
- Users can now completely remove themselves from the bridge, deleting all data.
- Bridge administrators can now be specified by adding an entry to the
'administrators' SQL table with your JID. These can force-unregister specific
users.
- Groupchats now support XEP-0313 Message Archive Management (MAM)!
- This uses the history stored in the sqlite database, as implemented in the
previous commits.
- The QUERY-ARCHIVE megafunction builds up a SQL query to get stuff out of the
database, in accordance with provided MAM + RSM parameters.
- Notably, various hacks are in here that need to be fixed.
- IQ 'set's are now processed, which means we needed to add a stub impl of
Schrödinger's Chat so people don't drop out of MUCs all of a sudden.
(Well, it just responds to every ping indiscriminately...)
- Oh also the new presence subscription stuff from earlier is borked.
- WARNING: This change is hilariously unsuitable for public instances. Don't run
this code in such an environment yet!
- Groupchat messages are now stored in the sqlite3 database, with the intent
to allow retrieval via MAM at a later date.
FIXME: You can't opt out of this though, which is a huge GDPR hole.
- You can also request ALL of your group chat history from whatsapp be fetched
and stored in the database (!!). This is a VERY resource-intensive operation,
as it involves spawning a metric crapton of threads, uploading a metric
crapton of historical media to the configured upload server, and writing a
metric crapton of messages into the database.
- At some point, the ability to invoke this will be severely limited to only
approved users. That point has not yet come, though.
- Additionally, the chat history request will abort if the connection it's
associated with dies. (You can just retry it, though.)
- Instead of taking a WhatsApp message and sending an XMPP one directly, we now
generate an XMPP-MESSAGE object, and have functions to deliver said object
and make said object from a WhatsApp message.
- This paves the way to storing said XMPP-MESSAGEs in a database, so we can
actually do message history / MAM. Yay!
- The memberlist refresh thing was supposed to delete all existing chat members
and then reinsert them all again.
- However, it didn't actually do the delete step. Big whoops.
- The "New or updated group chat" message is very spammy, and not even that
useful for being notified about a new chat you might have missed out on. We
no longer send that, but we do send something when you get a message for an
unknown groupchat.
- Presence subscription requests are now only sent if you have a chat open with
someone, because I got tired of receiving randomers' presence subscription
requests...
- Presence is now only ever sent in response to a presence probe, or when a user
updates their status or profile picture. You won't get presence from all
whatsapp contacts when the bridge connects any more.
- Getting hit with a massive backtrace is slightly overwhelming. It also is a
potential privacy issue, seeing as phone numbers (etc.) might be printed as
part of the call stack...
- Sometimes, the bridge can get wedged in a state where doing DB operations
returns SQLITE_MISUSE errors.
- I don't really know what causes this, but I'm hoping unconditionally resetting
the statements before they get used will ameliorate the problem?
- Additionally, the reset statements now aren't wrapped in an IGNORE-ERRORS,
meaning we might get error tracebacks that point to the actual problem now.
- This commit contains initial support for uploading and sending images natively
-- i.e. a WhatsApp image upload, instead of sending through a bare link.
- All images supported by the OPTICL library are supported; this is because WA
needs us to generate a jpeg thumbnail for the image (which can probably be
made smaller)
- Importantly, this functionality is currently a trivial DoS vector: simply send
a large file (or an infinite file, like time.gif) and it ALL gets downloaded
and buffered into RAM...!
- At least this is limited to users who have actually registered with the
bridge.
- Failing to send the image in any way results in an XMPP error getting reported
back to the sender.
- Also, the initial registration QR code upload thing now happens in another
thread.
- I don't think sending the user a crapton of presence when they send us some is
actually a good thing (and it's not doing any favours for my mobile data plan)
- Instead, let's let the user's server send us presence probes and reply to
those (?)
- The new PUT-WHATSAPP-MEDIA-FILE can encrypt and upload a blob of file data to
WhatsApp, and returns everything we'd need to make a protobuf message out of
that.
- However, important functionality is still absent: whatscl needs to gain some
support for sending protobuf messages for file uploads, and we'd also need to
integrate with some image/video libraries in order to do related processing
tasks.
- For example, uploading an image requires producing a JPEG thumbnail for it,
as well as figuring out its size.
- Uploading audio would require conversion to Opus.
- Maybe we could attempt to integrate with ffmpeg or something? This is likely
to get tricky.
- We now handle the :chat-modified event from whatscl, and re-fetch group
metadata when we receive it.
- Of course, this doesn't actually help for resources which are in the MUC at
update time; they'll need to reconnect until we have code to handle message
stubs properly.
- Chat metadata reception now generates a message to the user, both for ease of
debugging, and also to let them know that something happened to a groupchat
they're in which they would otherwise not be aware of at all. (A handy link is
also provided, if you missed the invite and want to join a MUC you're not in.)
- The new `refresh-chats` admin command forces a refresh of all group metadata
in one big go. (Also, the help text now displays the current version.)
- The "by" attribute of the <stanza-id/> element wasn't being set correctly (it
was just set to the localpart instead of the full JID, which is out of spec).
- This meant that Conversations (at least) wouldn't trust it, and would try to
use the origin-id instead, which couldn't be looked up when a chat marker was
sent to the bridge.
- This fixes that.
- (and adds another space to the put-url debug info)
- So the entity caps stuff didn't actually work, because we don't implement
XEP-0030 properly and don't honour the "node" attribute on the query.
- To hack around this, we misimplement XEP-0030 some more, and always return the
bridged user disco info list, even if they ask for the bare JID instead of the
full JID. That way, the caps stuff works as it should.
- The bridge should now support XEP-0115 Entity Capabilities, which actually
brings us into compliance with XEP-0085 Chat State Notifications (there were
clients, like Miranda NG, that wouldn't send to us without us actually doing
this XEP and advertising support in disco#info).
- We now handle the :disconnect message from whatscl, which lets us properly
deal with the connection being removed from the app or replaced with another
session. (Before, this would drop the connection, it'd loop and retry, and
then hit a status code error on login; this way is slightly cleaner and also
prevents the bridge getting into a war with another bridge.)
- The :ws-open whatscl event never existed. The bridge was doing just fine
not executing its handler, so it's just been removed. Future work might
improve the amount of spam sent to the user about the bridge state.
- We can now differentiate fatal login errors from other types of errors. For
non-fatal errors, we just send an informational warning message.
(Login errors kill the connection as before.)
- The child nodes of a given node can include both text nodes and
actual XML element nodes. Trying to pass a text node to something
like DOM:TAG-NAME signals an error condition.
- To avoid this, use DOM:ELEMENT-P and the CHILD-ELEMENTS helper
function to filter lists of nodes down to elements where elements
are expected.
- The user_contacts table contained an incorrect UNIQUE constraint
on wa_jid that meant only one user of the bridge could have any
given whatsapp user in their contacts.
- This is fixed in fixup_1.sql, which should be applied over schema.sql.
