This looks awesome, congrats for the hard work !
Nonetheless, as a non-coder only using Max, I have no idea how to use this…What steps would be required to have this working in Max ?
1 Like
Thanks!
Maybe it is more clear to tell the story from the beginning.
Step 1
Reverse engineering the sysex messages and write a low level library for encoding decoding.
This is libanalogrytm and although the contributions continue it is mostly complete.
Step 2
An easy to use and embed model of a Rytm project so developers can work with a data structure which represents the model of the project in actual device. And basically has access to the features listed here.
Step 3
Use the Max SDK to write an external which uses rytm-rs (The SDK which I’ve just announced) internally and provide a nicely designed API and user experience. If rytm-rs will be used for this project then median is a good choice. One can start this project and slowly develop it and release in between until it exposes the functionality of rytm-rs completely.
There is a second way, not to write an external but provide Neon bindings which exposes rytm-rs to Node.js. This way it can be used in a Node for Max object. This would enable a lot of developers in the Max/MSP community to create a patch with it and share it with users.
Step 1 and Step 2 is completed 
We’re now at Step 3 which is relatively the easiest part technically. It is more heavy on user experience but it might get better in time.
If someone wants to start that project I’d gladly contribute and if I find time and energy I might start it also.
It was a long answer but I think soon we’ll have something on our hands but I don’t know exactly when.
For you as a non-coder unfortunately you can not use it yet.
8 Likes
This is exciting!
1 Like
I was partially joking about the OS but has any attempt been made to decide the sysex OS files by anyone you know of?
I know people have done the machine drum so I’m just wondering if it’s even possible to have something similar on rytm one day (in 20 years? Lol)
Well done and a heartfelt thank you for putting the time into this.
2 Likes
I note that it’s been said of libanalogrytm:
So I’m now wondering what the steps might be (related to your steps above) to get something similar to your sdk working for, say, model:samples or digitakt.
My use case would be scripted changes or queries (using node js or ruby scripts) from a laptop.
I’m not familiar with many modern languages other than those two, but might dip my toes into Rust for this.
EDIT: Hmmm maybe I better figure out what I want to get out of this before diving into any coding:
I did this as a single individual and phew… it was a lot of labour of love. Also weeks of tedious reverse engineering and manual testing work has gone into it.
I’m going to answer this in a relatively long way to be informative since it might be interesting for the people who are interested in this thread.
For the ones who don’t want to read, yes it is possible but there is much more to it.
What exactly is sysex?
Think of sysex as a packaging around a hidden surprise present.
It always starts with 0xF0 continues with some metadata which are public to everyone such as manufacturer id and some more identifiers then you reach the present: “some data” and finish with 0xF7.
The some data part has a special encoding (7bit encoding) which is the second property which defines sysex. If you’re interested, read more about the standard structure here.
In short words opening the packaging of this surprise present and seeing what is inside is trivial.
- You read the header (meta data)
- You decode the data which is a common and known process
Now you have this surprise present in your hands. The only thing is, you don’t know what it is good for, you don’t recognize that object. That is the point where reverse engineering starts. From here on I’ll move on from this analogy and call this surprise present data.
Sysex is just a packaging over the data which can mean anything since it is just numbers.
Your shopping list, analog rytm pattern or Windows 10 
.
Sysex dumps
Fortunately when you have some context over what you’re receiving or querying and the data you’ve received is expressing some settings or parameters but unfortunately not documented you have some chance to figure out what it represents in a reasonable time frame with clever manual labor.
For example, when you know that the data represents an object in a rytm project, since you have a concept in your mind about what a rytm project contains you can make come up with a system to request parts of that project again and again while changing some values (simplifying) in the device per request and compare the response with the previous one to figure out what parts of the data change in this process. After some time the model of that data emerges. Then you may express that model with a programming language.
The key point is that the data comes with the sysex dumps which are already made available to the user has a defined context and we know roughly what it should contain.
To give you an example:
Here are two tracks in the data which comes from a pattern request from rytm. I’ve colored parts to figure out and focus to them individually. See that the colors repeat, probably you can even spot the track boundary.
The OS
When it comes to the OS it is a very different story.
If our surprise package contains an OS, it is not meant to be known since it contains a significant amount of intellectual property which is held by the company accumulated over many years.
Another thing is, while sysex dumps contain a list of parameters about an object i.e. pattern, the OS contains an executable file, a program for the device’s micro controller chip to execute.
A computer program is made of machine instructions which are again numbers like you see in the picture but when a processor (CPU or Micro controller) reads them it interprets them as actions, instructions. There are thousands if not hundred thousands of those instructions in an OS.
For the worse, those instructions are not written by humans, humans expressed the OS logic in a programming language such as C, C++ or Rust. Then they ordered the computer to translate it to those list of instructions which are called the machine code (processor type specific language). The process of translating the human readable programming language to the machine code is called compilation.
Enter decompilation
Now that we have the context. How can one disassemble the OS data to figure out what that means?
With the OS’s of today’s scale this would take years of error prone and tedious work to do manually. It is almost impossible.
This is why people created programs for decompilation and reverse engineering executable files such as ghidra.
But those programs are only a tool and a lot of creative guessing, trial and error, is necessary to decompile an OS completely which would take months.
I mean decompiling the rytm os and changing the start logo would be a fun task for a researcher and I think it is quite possible to do in a short time. On the other hand decompiling the whole OS and have control over it to develop it further or change it is a task of a completely different scale.
Imagine the functionality in Rytm from controlling the analog synth circuitry to sampling, effects to pattern management, menus to audio rauting, overbridge support… I can keep counting.
You need to understand how all and more of those components behave together in a high and low level way, also control or develop them without any documentation and no one to help you. It is a daunting task. You can come up as a team of individuals to do this but then maybe it might make sense to make your own product 
Legal concerns
Decompilation itself is not illegal. You can try to do this for research. On the other hand exposing Elektron’s internal source code would be. The reason is you’re sharing intellectual property which is not meant to be shared. Many programmers earned good amount of money over the years to create that property and actually Elektron owners have obtained that property in a very financially heavy way. That can get you in trouble.
After 10-20 years when the product becomes vintage… probably not that much but that is a speculation.
I’d say reverse engineering the OS would have been an interesting research project but doing something useful for the community with it such as a modified or alternative OS is extremely difficult, theoretically possible but practically wouldn’t worth it (opinion) at the time of this message.
17 Likes
Wow!
Any chance to get something like this for digitakt?
There was elk-herd Elk-Herd 3.1 beta for Digitakt OS 1.40
But mzero stopped updating it for the newer firmwares.
There are so many people waiting for a new update to be able to manage the patterns and projects on their digitakt…
1 Like
You don’t need to know Rust for this but dip your toes into Rust anyway it is lovely
This SDK is only for Rytm. Because it is the only Elektron device I have and use extensively and without the device in your hands it will not be possible to reverse engineer sysex properly.
For model:samples, if anyone didn’t do it before here is how to approach it.
Step 1
Study https://github.com/bsp2/libanalogrytm/blob/master/sysex.c and https://github.com/bsp2/libanalogrytm/blob/master/sysex.h.
Here you’ll find the query formats which might be portable across devices and sysex encoding and decoding helpers.
You can either write a node.js native module to encapsulate libanalogrytm or rewrite the useful parts of it in node again which is not a huge task.
Step 2
Try queries with the manufacturer id and device id of your device and analyze responses. That is where the reverse engineering starts. Reverse engineer the parts you’re interested and then test them with the device. If everything goes well write a nice interface over it for your use cases.
It should be possible but needs some work from someone who owns that device. If you have the time and interest and if you think it worths your time go for it.
5 Likes
I’m sure it is possible. Since I don’t own a digitakt probably I won’t do it but my code is open sourced with MIT license so if any other developer should adopt techniques from it or reuse part of the techniques there they’re open to do so.
If in the future, Elektron decides to document and expose their sysex format I think the developer community and 3rd party apps will flourish and you’ll have more choices.
But unfortunately now without documentation the entry barrier is a little high since it requires reverse engineering.
4 Likes
@alisomay Thanks for the explanation. I mistakenly thought from this
That some of the work for other Elektron devices had already been done.
It is partially true. I think if @mekohler tried this it should be true since I didn’t try it.
But you need to remember that is true for requesting the data.
If I was working in Elektron that is how I’d do it also, use the same header and a different machine id.
Although once you get the response you’ll receive the data structures relevant to the device you’re querying.
These are Pattern, Global, Kit, Settings and Sound for rytm but they might be different for the device you have. Even if they’re structured the same their inner structure, format and the parameters that they hold might have different ranges and structured differently.
To make use of this response you’d need to understand (reverse engineer) that format since it is not documented.
In short words probably the query format is the same but I’d be surprised the response format is also the same across all Elektron devices.
Once you make sense of the response format you can make interfaces to it to easily modify it, encode it to sysex and send it back to the device.
From my code base and libanalogrytm you can get some hints about how we deal with this process.
3 Likes
This is awesome.
1 Like
If anyone started to experiment with it I gave it a road test today and started to write a max external and found a few minor bugs. I’ll update it tomorrow
3 Likes
Good work, i don’t own a Rytm, so won’t benefit me. But kudos for putting in the hours and contributing to the community!
Unfortunately, this
Is probably why they won’t document their sysex. They lose control over their own APIs and have to maintain backwards compatability, support community asking questions, they’ll get hounded when stuff breaks, and they can’t make the changes they want/need for the feature they’re building.
Supporting apis for your own software is hard enough, but atleast they can say ”oh sorry, this machine version is higher than what the current overbridge supports, you must upgrade OB”.
6 Likes
Even if they were to publish versioned sysex info for every box, there would still be people wailing when stuff breaks.
I can quite see the downsides, for Elektron, in making this stuff public.
2 Likes
I agree with both of you to a certain extent but I’m sure that they have their own internal documentation and I still see benefit of them sharing it “as is” and say that we’ll not give support, use it taking the responsibility etc.
There is also another form of sharing. Sometimes companies share it by request. One time I have sent an e-mail to TC-Helicon for Voice Live Touch 1 and they sent me their sysex docs.
4 Likes
If anybody is interested to follow the max external development here is the repository.
I just started it and probably will not continue as fast as the SDK but eventually it will become something. I’m always open to contributions
4 Likes
I’d be intersted in a NodeJS module, if possible.
1 Like
Maybe I’d do that also, it is just some boring labor doing the neon bindings .
If somebody else would like to work on it before I do I’d be glad to contribute.
2 Likes