Summary
From time to time, WSPR transmitters note that they sometimes get reported in the wrong grid squares. Various possible causes are suspected including firmware bugs in the transmitter, software bugs in the decoder, general malaise in WSPRnet, and spurious noise decodes. In fact, the answer lies in the WSPR protocol itself, specifically in the Type 2 messages and the way that a 15-bit hash code links them to the real messages. There are in short, NO errors anywhere. Just an arguably over-used feature of the protocol which was never intended to be used that much.
Problem
This is not the first time I've seen this. Nor the first time I've heard of others complaining of it. But from time to time you see WSPR reports with your callsign, but NOT your correct location.
Here I was, trying to develop a "Virtual U3S" feature inside the QMX and QMX+ transceivers, to include all the functionality of the old Ultimate3S QRSS/WSPR beacon transmitter. I got various things hooked up including proper functionality of the GNSS (GPS) receiver, to set the real time clock. Which was a lot more fiddly than it should have been but that's another story. I set the supply voltage down a few volts so I measured about 1W power output to the OCFD from grid KM46. I left it running overnight, with local monitoring on my Pi400 with QMX+ transceiver to check DT (time offset) and local locator decodes.
Morning comes: The map looks something like this. Yep, that works...
So at first sight, it looks good... but then when you start looking at the actual reports, you notice something strange. From time to time, not often, but still, happening somewhat, there are reports from a totally different locator.
These reports have the wrong locator, they show grid subsquare CN87wr. (Full image here). They also have the wrong power level, +20 dBm instead of the +27 dBm I transmitted.
Note that this is the "old interface" WSPR report list. When you look at the WSPRnet spot page https://www.wsprnet.org/drupal/wsprnet/spots, near the bottom of it is a link called "link to old database interface" and this takes you to this page https://www.wsprnet.org/olddb which for whatever reason, I personally find preferable for viewing reports.
Now if you look at the iPhone WSPR application "WSPR Watch", which plots the start and end point of each WSPR report for the callsign (rather than the WSPRnet map which just plots the last known transmitter locator) you can actually see these reports coming from grid subsquare CN87wr. Full image here. (I hasten to add, not my iPhone per se... I am a loyal Samsung user).
Or you could accidentally have a situation where the last uploaded spot to WSPRnet had the false locator, in which case all your receivers appear to go back to the false locator:
Confusing evidence
So now the question, where is this spurious locator coming from?
Noise decode?
Spurious decodes of noise do occur, but are rare. The WSPR protocol has a large amount of Forward Error Correction and the decoding of "illegal" things happens rarely. Rarely, but not never.
For example as I write this, I have on my QMX+ with Pi400 that I was using for local monitoring, a decode of DJ8XIZ, in grid AN92, at +60dBm (1,000 Watts)... That radio currently has nothing connected to its antenna port, not even a dummy load. DJ8XIZ doesn't seem to exist (not on QRZ.com - which is not conclusive - but also Google shows up nothing at all). And AN92 is in the Northern Pacific ocean somewhere between Hawaii and Alaska. Using 1,000W for WSPR transmissions would be a little unusual too. So it's just a noise decode.
BUT - it gets vanishingly unlikely, that the same locator and power level, would be identically decoded by multiple different stations in the same timeslot. A random noise decode by someone somewhere is one thing. But more than one random decode of the SAME information seems impossible... So noise decodes isn't a plausible explanation.
Firmware problem?
Now what was generating the WSPR here was my new firmware for the "Virtual U3S". Maybe I had some bug in my firmware... But then again, when I am decoding the signals locally on my Pi400 with QMX+ and it's 10-inch HDMI monitor, why would I not get those incorrect locators locally too?
Yet, no local decodes with the incorrect locator are seen.
So again, it doesn't seem a problem with my actual transmission.
Timing?
Now the plot thickens because when you examine the spots, and you know that the QMX+ transmitter (new "Virtual U3S" mode) is transmitting on a fixed cycle every 10 minutes, so all the transmissions occur at say, :04, :14, :24 etc minutes past the hour - then you note that the reports with the incorrect locators sometimes come in time slots when the QMX+ was not even transmitting!
But even this is not conclusive, because there are also plenty of examples of correct decodes being published in the wrong timeslot - and this could presumably occur if the WSPR monitoring station had issues with his internet connection and it took a long time to create the upload, or perhaps the time on his PC was off by a fairly exact multiple of 2 minutes, or who knows what could happen. It does also happen, you can find this in the data too. It usually only is 2 minutes behind the real transmission.
Some time slots simultaneously have decodes of both my correct locator KM46, AND the incorrect one CN87wr. So how could my transmitter be transmitting both a correct callsign and an incorrect one... at the SAME time? Well actually even that IS possible, if you have a firmware bug - for example I had one instance where I changed my callsign configuration to my US callsign AF7BF. Due to a firmware bug doing something like not completely resetting all bits of the message, the next local decode (and indeed some remote station decodes) had some G0UPL and some AF7BF in them! Which is really weird...
But then - I observed that the spurious decodes came in, not only on incorrect time slots when my QMX+ wasn't transmitting; but I had even switched it off completely, 20 minutes earlier, while I worked on some firmware changes!
So these spurious decodes really could NOT originate from my QMX+. But in that case, where do they come from?
Pirates?
The other inescapable fact is that if I was not transmitting WSPR for some time, I was doing other stuff with my radio for a few weeks for example, then there were no incorrect decodes. Which is as you'd expect. These kinds of erroneous decodes only started on days (and hours) when I was actually transmitting WSPR! So it was *something* I was doing, transmitting WSPR, that was causing the incorrect reports. Not false noise decodes, not firmware problems in my transmitter. But nevertheless something done by me.
OR - could it be a pirate? Someone somewhere who really doesn't like me much? Who has some alert set up and when he sees my callsign on WSPR, thinks he'll just transmit something also, using my callsign, with a different locator and power, just to mess up my life a bit? Seems unlikely - yet faced with all the above evidence, what other conclusion could be drawn? If all the possibilities have been eliminated, except the most unlikely one, then the unlikely one looks less unattractive.
AND - look at the maps above. It appears all the spurious decodes cluster around East coast US. And a couple of Canadians, usually. Whereas when I transmitted, I got 100+ reports from each transmission, the majority from Europe but further afield too, from one side of the Globe to the other; when "the pirate" transmitted, the reports are mostly from east coast US and usually less than 10 per transmission.
But do we really want to rely on an explanation involving a Pirate? Blaming a pirate feels a bit like giving up. But how else could I explain the fact that nothing could possibly indicate I transmitted this myself; yet, the spurious reports only show up if I've actually been transmitting?
First real clue...
Now it occurred to me, as so often happens, in the middle of the night or weekend or some other time when radio was not actively being thought about in my brain - that this might be something to do with the 15-bit hash codes involved Type 2 WSPR transmissions. I resolved to go to the computer at the earliest opportunity, and check the "incorrect reports", and see first, if the stations copying "the pirate" had also copied me. If so, that would be a strong clue.
Sure enough: 100% of the reports of "the pirate", had also copied me before that. Strong clue!
Which also somewhat explained why the pirate reporters tended to cluster in the US Eastern states. Because these are the reporters which heard both me, far away in SW Turkey, AND the pirate, who is on the US West coast. Of course lots of other stations across US heard him too, but they were less likely to have also copied ME unless they were Eastern US states.
What are type 2 WSPR messages and hash codes?
A usual WSPR message sends 50 bits of information, encoded into 162 symbols, each of which is one of four tones. The transmission duration of each symbol is 256 / 375'ths of a second, so 162 of them take a little under 1 minute 52 seconds to send, hence the idea that WSPR transmissions are 2 minutes long which is approximately correct. The four tones are spaced 375 / 256'ths of a Hz, in other words 1.46 Hz, beautiful symmetry all around.
The 50 bits of information contain the Callsign, Locator, and power level.
G0UPL KM46 27
But with limitations!
- Callsign can only be a normal 4, 5 or 6 character callsign. You can't send something like TA4/G0UPL or G0UPL/P.
- The grid is only the 4-character Maidenhead grid. You can't send a higher resolution grid subsquare.
- The power level is restricted to 00, 03, 07, 10, 13 etc. up to +60 dBm. You can't send say, +11 dBm.
BUT - there are two types of special WSPR encoding. One can send a non-normal callsign. Which means, a callsign with a prefix, or a suffix obeying certain rules (but not both). The other type can send a 6-character Maidenhead grid subsquare.
Of course, we only have 50 bits of data the WSPR protocol can actually send. Something has to go... We can't send a bigger callsign, or a bigger locator, without saving some bits somewhere else. So the way WSPR handles this is something called the hash-code. The standard callsign (4-6 characters) is somehow mapped, or compressed, we could say, into a 15-bit binary number. That is in other words, a decimal number between 0 and 32,767. But since there are many more than 32,768 possible callsigns in the world, what this means is that multiple callsigns can have the SAME hash code. This is a crucal point, remember it...
The special WSPR transmission contains this 15-bit hash code, which is a kind of compressed version of the callsign; then the extra information such as the longer Maidenhead grid subsquare, and the power level as usual.
Now, any station transmitting this additional information (complicated callsign, or longer 6-character Maidenhead grid subsquare), transmits TWO different WSPR messages, in two different 2-minute slots.
- Normal WSPR message, such as G0UPL KM46 27, the usual callsign + 4-char locator + power level
- Special WSPR message, for example containing the 6-character grid subsquare; and the hash code, and power level
The special transmission always contains the compressed callsign, which is the 15-bit hash code; together with the longer information (callsign prefix/suffix, or 6-character Maidenhead grid subsquare), and the power level is included as usual.
Every WSPR receiving station, upon decoding any normal WSPR transmission, takes that callsign and calculates its 15-bit hash code (or "compressed" version of the callsign), and stores it in a hash table. The hash table can contain up to 32,768 entries, one for each possible permutation of the 15-bits of binary. It's a mapping of these 32,768 possible hash values, to actual callsigns that the specific WSPR receiving station has received.
Now when a "special" WSPR transmission type is received by that same WSPR receiving station, it decodes the 15-bit hash code from that special WSPR transmission, and looks in its "hash table" to see if it has previously received that hash code, from a normal WSPR transmission. This is how the WSJT-X software (and others) match up the "compressed" version of the callsign which is the hash code, to the full callsign. It's whatever it finds stored in its hash table.
Sometimes if you are a WSPR decoding station, you might see a decode on your screen with a callsign like <...>. What this means, is that your station decoded a special WSPR transmission, with a hash code, for which your station has no entry in its stored hash table, therefore it cannot decompress that hash code into a full callsign, and since it does not know the callsign, it just writes <...> for now.
For better or for worse (more likely), this is all controlled by a checkbox on WSJT-X when you are in WSPR mode, called "Prefer type 1 messages, as shown in this screenshot below:
Hovering over this checkbox helpfully (somewhat) provides the following explanation:
"6 digit locators cause 2 different messages to be sent, the second contains the full locator but only a hashed callsign, other stations must have decoded the first once before they can decode your call in the second. Check this option to only send 4 digit locators if it will avoid the two message protocol".
This pretty cryptic explanation basically means all of what I explained above. "Check this option to only send 4 digit locators if it will avoid the two message protocol"... eh? What does that mean? Well what it really means is that if you ticked the checkbox, your WSJT-X will only send normal WSPR messages (simple callsign, simple 4-character grid locator square, and power). If you left the checkbox unchecked, then if you specified in the WSPR settings your callsign as being complicated (having a prefix or suffix); OR, if you specified your location as 6-character grid subsquare, then your WSJT-X will keep sending TWO messages. The second, will contain the hashed callsign and be linked up to the first, using the hash code (consider it a compressed callsign kind of). And the other stations can only associate the two transmissions together correctly, if they have received and decoded both transmissions from you.
BUT - what about the fact that the 15-bit hash is a many-to-one mapping? In other words, multiple callsigns can produce the same hash code?
More on hash codes
A word about how the hash code is generated. I have some C code which does it, which I cannot remember where on Earth I got it from, more than a decade ago; or even if I made it up from the WSPR source code, I can't recall, BAD me, I neglected to note where it came from, in my code comments.
It turns out to be a lot less easy to make this in an Excel sheet, than in a C program; and am not clever enough on web stuff, to write it in Javascript on this page, or something like that. Anyway eventually I managed to make it work in Excel. I am not sure if this is some official mathematical process, or if it's just a number of steps of mangling that eventually results in a pseudo-random relationship between 15-bit codes and callsigns, with fairly even distribution across possible callsigns.
To generate the hash code, we start with a constant value. Which happens to be the 32-bit number expressed in hexadecimal format as: 0xDEADBEEF. Dead Beef? So someone with a sense of humor thought of this to start with. Joe K1JT? Anyway. Then that gets decimal 146 added to it - which in hexadecimal is the unremarkable 0x92 (for who knows what reason - Joe won the Wolf Prize in Physics in 1992 and the Nobel Prize in Physics in 1993, would any of that have motivated the year in hexadecimal 92, who knows, that's a speculation too far), and the length of the callsign, e.g. 5 for G0UPL. This produces three constants a b c, to which we add the various multiples of the ASCII codes of the callsign characters. Then we go through 7 stages of mangling things up, and take the least significant 15 bits of the final answer to be our resulting end result, the hash code.
Readers may now download THIS EXCEL FILE which demonstrates this hash encoding process. You can enter your callsign in cell C1 and see the 15-bit hash code result generated in cell C2.
Putting all the pieces of the jigsaw together at last
With all this information and thought process, I can create a theory, with the features
- There is no malicious pirate who hates me (maybe there are lots of people who hate me, but they aren't WSPR pirates);
- There is just a regular dude, henceforth to be referred to as The Regular Dude, who has entered CN87wr as his 6-character grid subsquare in his WSJT-X settings;
- The Regular Dude, has also NOT clicked the checkbox in his WJST-X which would force it to only send regular WSPR messages;
- Therefore The Regular Dude's station sends two types of messages, the special one containing his 6-character grid subsquare CN87wr and his 15-bit callsign hash code;
- The Regular Dude has the SAME 15-bit hash code result as G0UPL. We can call this a "hash collision";
- Several WSPR stations received MY regular WSPR transmission, which put my hash code and callsign in their WSPR hash table file
- Then those WSPR stations received The Regular Dude's "special" WSPR transmission, and looked in their hash table and found G0UPL for it, then uploaded the spot for that G0UPL CN87wr 20 (where 20 is the actual power setting of The Regular Dude)
Think about it, this then fits ALL the observations.
If I am never on air, or not on air for a long period, then nobody has G0UPL in their hash table. But as soon as I go on air, a whole bunch of WSPR receiving stations all over the world, receive my signal, and they immediately put my hash code in their hash table, mapping to my callsign G0UPL. Then there's The Regular Dude, who happens to be transmitting the non-type-1 messages, or "special" WSPR messages, sending his 6-character Maidenhead locator CN87wr.
The Regular Dude's power level setting is 20 (+20dBm, 0.1W), which is much lower than I was using. And quite likely, there are some WSPR receiving stations far from The Regular Dude, who didn't copy his every utterance, but had happened to have copied me, therefore had G0UPL in their hash table; and when they happened to receive The Regular Dude's special transmission with the hash code, of course they found G0UPL in their hash code and faithfully reported this mis-information to WSPRnet.
And if I stop transmitting, go off air? Then as I watch, over the next hour or two the incorrect reports keep coming in, even though my radio is switched off, because some WSPR stations keep receiving The Regular Dude's special transmissions but haven't yet received his usual transmission, which would kick me (G0UPL) out of their hash table and replace me with HIS callsign. But gradually over an hour or two, the frequency of occurrence of these spurious reports, The Regular Dude masquerading as The G0UPL, start to diminish. This is also natural and expected - since these WSPR receiving stations that had me in their hash table, will likely eventually receive The Regular Dude's normal WSPR transmissions, replacing me in the WSPR receiving station's hash table.
In other words, when I (G0UPL) transmit, there is a hash collision, corrupting WSPRnet by mis-reporting another station's (The Regular Dude) reports, attributing them to G0UPL As soon as I stop transmitting, the hash collision gradually gets purged out of the hash table files of WSJT-X on WSPR receiving stations around the world; and the mis-reporting accordingly dies out by itself.
Final proof, and who IS The Regular Dude?
Now it occurred to me, The Regular Dude does not need to remain forever anonymous. There's as I said, a many-to-one mapping of WSPR callsigns, to 15-bit hash tables. But I can find this guy, The Regular Dude.
To find him, I waited patiently, watching my WSPR transmissions, when propagation to North America was open for me on 20m from my SW Turkey QTH, waiting for the erroneous reports to pop up in WSPRnet. As soon as I saw a bad report, placing me in grid subsquare CN87wr, I quickly quickly checked this page:
https://www.wsprnet.org/olddb?mode=html&band=20&limit=9999&findcall=&findreporter=&sort=date
which lists all the last 9,999 (the maximum allowed) WSPR reports on 20m. You have to be FAST because 9,999 reports is less than 10 minutes' worth of WSPR action on the 20m band, so the evidence quickly evaporates. Well not really, it's still there, in the WSPR database; and you can in fact, download the WSPR database month by month, but those are gigantic files. OR, there are other WSPR websites too now, and maybe some of them provide better ways to search for information for example by location not by callsign, etc. So the evidence didn't really evaporate, it just gets harder to find it. And I like an easy way.
So anyway - as soon as I see that page of the last 9,999 WSPR reports on 20m, all of them (no filter on the callsign), I know The Regular Dude is probably in there somewhere. Now I can just Ctrl-F and search for CN87wr. And I find him! KI7E.
The Regular Dude is KI7E.
Paul KI7E, https://www.qrz.com/db/KI7E who is even a QCX+ owner!
His locator is of course CN87wr because that's what I searched for. And I can see also that his time of transmission matches the same time that there was a spurious report of G0UPL CN87wr 20. AND, his power level is also 20dBm (0.1W). So everything matches. We can even look him up on QRZ.com see https://www.qrz.com/db/KI7E and it says he dabbles a bit in WSJT-X modes and has even done some high altitude ballooning, so yeah, he really is a DUDE.
Wait - I made a leap of faith that there wasn't some other Regular Dude also in grid locator subsquare CN87wr (which after all, is something like an area of 5-10 square miles, it isn't impossible), also transmitting WSPR at that same time and with +20dBm. It isn't impossible.
So lt's check the hash code using my horrible ugly spreadsheet! You know what we find? KI7E and G0UPL both have the same hash code: decimal 11,536.
Proved!
Conclusion and recommendations
- There may be other cases. But certainly this time (and I believe for many similar complaints) the spurious locators are
- NOT a WSPRnet glitch
- NOT a noise decode
- NOT a pirate
- NOT a transmitter firmware issue
- NOT a WSJT-X bug
- Just a simple case of a feature of the WSPR protocol, and hash collission
- Please please please, unless you absolutely HAVE to do it, for example if you need to track yourself moving (resolution a few km, the Maidenhead grid subsquare) or have GOT to send some weird callsign pre- or suffix, please please make sure your "Prefer Type 1 messages" checkbox is ticked!
And remember:
- Even if you ticked that checkbox yourself, other people can still cause interference (erroneous reports) to you via hash collision;
- If you tick that checkbox, you will still be able to cause interference to other transmitting station's reports, who did not tick it via hash collision
- If you tick that checkbox, you CANNOT cause interference to any other transmitting station's reports, who also ticks the checkbox
- If you are a station who does not tick that checkbox, then bear in mind you are going to lose a whole bunch of your reports, to people who haven't received both your transmission types, or have received someone else's transmissions via hash collision; so you will reduce the effectiveness of your WSPR operations, as well as increase "interference".
In my opinion, it would be better if WSJT-X had HIDDEN this setting in some advanced configuration settings menu, with a lot clearer warning about the consequences, than the current floating help popup!