10/10/2016

Torrent-tracking across networks

I have never given lots of thought to torrents, mostly it has for me been a thing to use to get hold of linux distro images. Although at the end of a small video production project (for Var GladSpexarna), and given the very modest server means available, the size of the file, and the number of potential recipients, BitTorrent, with its decentralized distribution seemed to be the right method.

Earlier years, before I was in charge, DVDs were burned, but the availability of newer cameras with higher resolutions have obsoleted that method, and even if bluray could possibly be used, my personal opinion is that, in the age of streaming and digital storefronts, optical media is on the way out. I am also not a fan of the prospect of being the one burning all these discs.

Torrents were given the OK from above, and a friend set up a small tracker. Since the aim is internal distribution, that was opted for, rather than using one of the various public trackers. The torrent is private as well, and the files are encrypted. I would have run a tracker myself, if not for the fact that I seemed to be unable to set up any sort of external visibility on my machines. I wrote that off as being yet another dimension of failure of the cheap 3G modem that is my connection to the internet. Other things required my focus.

Except it wasn't, or something changed. I don't know, but for some reason I had another go at it, and found out about hairpinning, and how routers may or may not support it. So I'm back in the running-my-own-server game at last.

The server now includes a torrent tracker. Because I want to run torrent tracker. The tracker is opentracker, the same as my friend uses, with a few modifications by yours truly, and those are the reasons I'm writing this post.

The problem I ran into with the tracker is that if I run the client with the initial seed of a torrent on the same host as the tracker itself, the address associated with that client will be 127.0.0.1, localhost. When another client asks for peers, the tracker will respond that there is one at 127.0.0.1.
This will of course not work.
The same thing goes for when there are clients running on the same local net as the tracker, say 10.0.0.0/8, and a client on a machine somewhere on the internet asks for peers.

At some point in time, the clients on local addresses will update their peer list, and thus ask the tracker again, and the tracker may inform them of the data-starved machine on the internet and initiate a connection.

The problem is succinctly put: "A machine cannot be the initiator of a connection between it and a machine on a local net behind a gateway, by addressing that machine by its local address." Kind of obvious when put that way.

Some of these problems can in some cases be fixed by having the tracker and client support setting a chosen IP address, rather than use the address from the connection between client and tracker. This was not a viable solution for me for two reasons:

  1. The torrent client I use, especially for running as a daemon is transmission, which does not support setting a custom IP field.
  2. If I could set an IP address, and I set it to the external address of my modem/gateway, I would not be able to connect to that client from my local network, due to lack of support for hairpinning.
The solution I opted for, partially to see if I could do it, partially since it seemed to me to be the easiest way to have everything I wanted without compromise, was to modify the tracker.
I present to you, my variant of opentracker. The modifications I have done, amount to doing an address translation of the peer list based on from where the request comes. The translations are rule-based so any translation expressible by the config file format should be possible, but the assumed use case is to e.g. translate local addresses to the gateway's external address when a machine outside the local net is asking.

25/09/2016

Reverse engineering the Puxing PX888K programming format (3: Patch submission and coding style)

The driver I have written has now been accepted by the CHIRP project repository and while I have made sure that it passes the testing suite of the project, as well as that the code adheres to the pep8 guidelines, now the real trials can begin: Actual use. But until the bug reports start rolling in I'm going to feel invincible for a bit. Click-click.

I may have one or two things to say about pep8 though, mostly regarding the line length limit. After a few levels of indentation it starts to put limits of what one can put in string literals, string literals which may be error messages which one would want to understand, and be able to grep for. The Linux kernel coding style documentation has my back on this, or rather, it's probably fair to say that it's the actual source of my thoughts on the matter.

Another thing I think is detrimental when hard-lining at 80 characters is when calling the same or a similar function a lot of times with slightly altered parameters, such as populating a menu.

From a readability standpoint, I think that something like this (please disregard the lazy table formatting):
func_0(param00,param01,param02)
func_1(long_name_param10,param11,param12)
func_2(param20,long_name_param21,param22)
func_3(param30,param31,long_name_param32)

...is more readable than this:
func_0(param00,
param01,
param02)
func_1(long_name_param10,
param11,
param12)
func_2(param20,
long_name_param21,
param02)
func_3(param30,
param31,
long_name_param32)

...as long as the functions are related and have similar signatures. The first params are all in the same column, as are the seconds, and so on. But if some of the params are string litterals you will soon be at the end of your line length allowance, especially since you're likely to do it at some level of indentation.
At some point it would make more sense to move the data away from the code, and instead just loop over a two-dimensional array or a map, calling the desired functions. I guess that would be the standard answer to these concerns actually: You are not supposed to have that many magic numbers (or magic strings as it were) in your code! I don't disagree with this, but it does depend on whether or not the rest of the project uses this method.

Another point is how much code can be shown at once, the first example takes four rows of my editor to display, allowing me to see more code above and below it, compared to the second example that requires twelve rows. And this in the days of widescreen monitors too. Of course, if you have one of those tiltable monitors I envy you, although all of the one's I've used have had a narrow enough viewing angle that my left and right eye ended up seeing slightly different colors which irritated me to no end.

13/09/2016

Reverse engineering the Puxing PX888K programming format (2: Making sense of the data)

I have found out that the reverse engineering was the easy part. It should however come as no surprise, since if communications protocol and data format are simple, and they are, the whole thing becomes a iterative process of changing a single setting to a new value, and diff the old communications dump against the new, to see what bits have changed.

Using jpnevulator to listen in on the data transfer between the stock software and the radio, the program can listen to multiple serial ports at the same time, and neatly display which port sends what when. This way the back and forth between the stock software and the radio can be easily viewed and the protocol is quickly figured out. There is a short initial handshake in the beginning, followed by either reading or writing the whole image in chunks of 64 bytes.
The protocol, but just as much an image to break the text
The next step was to implement a few scripts that could play the part of the radio, but dump the memory written to an image file, instead of writing to the radio each time, and reverse engineer from the snooped data which contains the communications commands which, we are no longer interested in. The script can also read from an image file, making it possible to poke at the bits and see what changes in the stock software. Functions for reading and writing to the radio are also implemented, which is useful since the stock software does not cover all settings available on the radio. The scripts are available on github.

After a data communications failure, it also became apparent that there are features not available in the stock software I had, as well as the radio: Something went wrong and I found myself with a radio with a locked keypad. It turns out that this is a feature rather than a bug, and a newer version of the stock software includes the possibility of changing this setting. The new software version also makes the feature Relay without disable tail available, which I have no idea what it does, and I can't find it documented anywhere. The closest thing I have found when searching is that it may enable or disable Squelch tail elimination, but I don't have the equipment to test this feature currently.

Another setting that is poorly documented is Wait time which can be set to 0.3, 0.5 or any integer in the range 1-12. I do not know what it is waiting for. It does not seem to be any of backlight timeout, keypad lock timeout, or any delay regarding when PTT ID or other data signals are sent. Possibly it could have something to do with when the battery save functionality sets in, but I don't know how to test this reliably. A bit of keyword searching lead me to a French forum which hinted at, and is corroborated with my own following tests, that it is how long it waits in before it resumes scanning, when using carrier based scanning.

A bigger challenge, the "but" continuation of the "easy" in the beginning of this text is writing a CHIRP plugin. It would be easy enough to pour data between a csv file tailored for this radio, and this radio only, and the binary image would be easy. It is a different matter entirely to fit this into a system that also needs to cover how other radios work, and what their features are. But the point of doing this is to make it possible to copy settings between radios that would otherwise be each programmed by their own software platform, all incompatible with each other. I can certainly get behind that kind of endeavor. Another part is that doing this project with the goal of writing a CHIRP plugin is probably the best way to have it reach other users of this particular model and have them benefit.

10/09/2016

Reverse engineering the Puxing PX888K programming format (1: The joy of soldering)

A while ago I bought a Puxing PX888K VHF+UHF radio HT. Like for many other radios, it's possible to change the configuration from a computer. Like for many others, the program is severely lacking in polish and only available for windows. To its defense though, it works more or less as good (or bad) running it through wine on Linux, as it does on windows. But it offers nothing like export/import CSV, or even though the channel list looks like an Excel sheet, it is not possible to copy or paste rows.

The perhaps default open-source radio programming software distribution CHIRP does not at this time support this radio model. Maybe, if I'm successful in my work, that can be changed.

First things first. To decode the data, I need to get the data. To get the data I need to listen on the communication wires, so a junction box was built. Given that I'm building with what I find in my parts bins, rather than routing a proper board and ordering parts, the final result does perhaps look more grown than designed. ...And that's not really false marketing.

junction box and schematic junction box back
The amount of switches comes from the fact that one of the serial cables I use for snooping is straight, while the other is a null modem cable, as I wrote, I'm working with what I have here.
Also, I want to keep my options open.

Since the radio is programmed with 5V signals, and the snooping ports work at RS232 levels, a level converter is needed. Again, lots of switches for connectivity options, and mixing SMT and through-hole devices oh my.

level converter and
simplified schematic
level converter back
A quick gander with the logic analyzer and a few attempts to read shows that the data is transmitted as regular serial 9600,8,N,1

action shot
To be continued...

06/07/2016

More DMX512 Protocol Interfacing

This spring gave me opportunity to revisit a few old projects, the led bars previously documented, and the hardware I designed earlier for interfacing with DMX512 equipment.