can not register account with GMRS call sign on mygmrs.com

[Guest WRMM237]

Hello mygmrs.com'ers!

I've been trying to register here for 2 weeks. What do I need to do? (see pic)

[Guest wrmn710]

I have the same problem.  My call sign was issued on 5/8.  based on things I've read on the forum, I guess keep trying.

[rdunajewski]

We have a help desk ticket into the FCC about this. So some licenses are not getting ingested into our system, but others are okay. The data is there in the files they give us, but there's a critical problem preventing it that I believe they should fix.

The FCC provides two types of database updates to us, a weekly "full" database copy, and a daily transaction file that contains only updates from the previous day (or 2 days if they're slower to update it).

We use both, we use the full database to populate up until Sunday, then catch up to the current day by playing back the daily files since Sunday until we're caught up with what data they provide to us (which again, is still at least a day behind what you can search on their site).

These files are a text format where each data field is separated by a pipe "|" character, and a new line for each new record. It's similar to CSV (comma-separated values) format but uses the pipe character instead since it's less common in a data field than a comma is. Whenever you have this separator character in the data set, you need to escape it somehow to tell the parser to ignore it as a field separator, usually with a backslash in front like "\|" or double quotes around the entire field. 

There is a certain license in the FCC system that appeared on April 27, 2021 for WRML255, that caused havoc in the data. There is a field for the title of the person signing the application for the GMRS license, and in this case the applicant entered "Owner|Operator" in this field. Here's what the FCC record shows:

HD|4441017|0009519764||WRML255|A|ZA|04/27/2021|04/27/2031||||||||||N||||||||||N||Robert|A|Cecchino||Owner|Operator||||||||04/27/2021|04/27/2021|||||||||||

Note how the pipe character is used to tell each field apart, and the number of fields must be predictable for us to know that field 3 or field 9 maps to a given field like a name or expiration date. The pipe character in "Owner|Operator" is not escaped by the FCC, so our software tries to split it into 2 fields, and keeps going down the row (but now we're mapping the rest of the fields off by one place), until we get to the last field. We expect 55 fields in each row, and we just detected a 56th field. Uh oh, parsing error! 

The FCC should escape the pipe character so it shows like so, as one example:

HD|4441017|0009519764||WRML255|A|ZA|04/27/2021|04/27/2031||||||||||N||||||||||N||Robert|A|Cecchino||Owner\|Operator||||||||04/27/2021|04/27/2021|||||||||||

Then we'll ignore it as a separator, and correctly detect 55 fields for this row.

This case only appeared in one daily file in the past, so the next day it was okay and maybe a handful of licenses were affected. But when the full database comes around each week, this offending case is always there and it causes us to skip everything after this row. Your callsign is after this license, so you must not have been in a daily file for one reason or another (either the FCC missed it, or we couldn't fetch it on that day). You'll never appear in our system until this parsing error is fixed either on our end (with a real hack, very messy) or on the FCC's end, which I think is the correct place since this is a broken file, anyone else parsing it would have the same issue unless they put in their own hack to fix it. A hack is definitely not the right way to fix it from an engineering standpoint.

If the FCC is able to commit to fixing it, perfect. It'll resolve itself once they fix the weekly database file. If they don't, then we have to find a reliable way to fix this and future cases, which is really suboptimal.

[Guest WRMM237]

Thanks for the detailed explanation @rdunajewski

As a fellow data parser, I feel your pain.

HD|4442825|0009533178||WRMM237|A|ZA|05/01/2021|05/01/2031||||||||||N||||||||||N||Christopher|J|Sotak||||||||||05/01/2021|05/01/2021|||||||||||

I'll put this in my pipe and smoke it (in the meantime)

https://stackoverflow.com/questions/492090/least-used-delimiter-character-in-normal-text-ascii-128

 

[mbrun]

We have a help desk ticket into the FCC about this. So some licenses are not getting ingested into our system, but others are okay. The data is there in the files they give us, but there's a critical problem preventing it that I believe they should fix.
The FCC provides two types of database updates to us, a weekly "full" database copy, and a daily transaction file that contains only updates from the previous day (or 2 days if they're slower to update it).
We use both, we use the full database to populate up until Sunday, then catch up to the current day by playing back the daily files since Sunday until we're caught up with what data they provide to us (which again, is still at least a day behind what you can search on their site).
These files are a text format where each data field is separated by a pipe "|" character, and a new line for each new record. It's similar to CSV (comma-separated values) format but uses the pipe character instead since it's less common in a data field than a comma is. Whenever you have this separator character in the data set, you need to escape it somehow to tell the parser to ignore it as a field separator, usually with a backslash in front like "\|" or double quotes around the entire field. 
There is a certain license in the FCC system that appeared on April 27, 2021 for WRML255, that caused havoc in the data. There is a field for the title of the person signing the application for the GMRS license, and in this case the applicant entered "Owner|Operator" in this field. Here's what the FCC record shows:
HD|4441017|0009519764||WRML255|A|ZA|04/27/2021|04/27/2031||||||||||N||||||||||N||Robert|A|Cecchino||Owner|Operator||||||||04/27/2021|04/27/2021|||||||||||
Note how the pipe character is used to tell each field apart, and the number of fields must be predictable for us to know that field 3 or field 9 maps to a given field like a name or expiration date. The pipe character in "Owner|Operator" is not escaped by the FCC, so our software tries to split it into 2 fields, and keeps going down the row (but now we're mapping the rest of the fields off by one place), until we get to the last field. We expect 55 fields in each row, and we just detected a 56th field. Uh oh, parsing error! 
The FCC should escape the pipe character so it shows like so, as one example:
HD|4441017|0009519764||WRML255|A|ZA|04/27/2021|04/27/2031||||||||||N||||||||||N||Robert|A|Cecchino||Owner\|Operator||||||||04/27/2021|04/27/2021|||||||||||
Then we'll ignore it as a separator, and correctly detect 55 fields for this row.
This case only appeared in one daily file in the past, so the next day it was okay and maybe a handful of licenses were affected. But when the full database comes around each week, this offending case is always there and it causes us to skip everything after this row. Your callsign is after this license, so you must not have been in a daily file for one reason or another (either the FCC missed it, or we couldn't fetch it on that day). You'll never appear in our system until this parsing error is fixed either on our end (with a real hack, very messy) or on the FCC's end, which I think is the correct place since this is a broken file, anyone else parsing it would have the same issue unless they put in their own hack to fix it. A hack is definitely not the right way to fix it from an engineering standpoint.
If the FCC is able to commit to fixing it, perfect. It'll resolve itself once they fix the weekly database file. If they don't, then we have to find a reliable way to fix this and future cases, which is really suboptimal.

You know, fixing that issue will require congressional approval.


Michael
WRHS965
KE8PLM

[Citizen]

Rich,

Would you be able to (temporarily, until fixed by the FCC) pre-scan the pipe-delimited file(s) looking for the Owner|Operator string?  You could use a batch mode WHS or VBA script (or many such other tools) to first scan, then if found, change the string to Owner\|Operator , before inputting into the call-sign database.  This is something I have done numerous times in my former job as an applications systems analyst.  I can offer help if needed, or possibly write it for you.

 

Thomas

[rdunajewski]

I'd rather not put in hacks like that, it adds a whole step to the process that is unnecessary. The output file is technically broken, so the fix needs to be done by the FCC. If they for some reason refuse, then we'll have to get more creative with it.

It's also not a one-time fix, as this condition happens every week, and it won't be immune from another pipe character appearing in a field in the future, only this one specific case.

Hopefully we hear back from them on Monday so we know the game plan.

[Citizen]

2 hours ago, rdunajewski said:

[snip] The output file is technically broken, so the fix needs to be done by the FCC. If they for some reason refuse, then we'll have to get more creative with it.

Hopefully we hear back from them on Monday so we know the game plan.

Of course, makes sense to wait a bit for them to fix it.

...

[tjharrell]

Gotta love free-form user input.

I parse these things too, and what I do is write out all the parsing errors to a separate error file for manual review.  Those records go into a two-column database table with the original record as received from the FCC in one column, and a modified parseable record in the second.  Then the next time the record is parsed, if there are any parsing errors then the record is used as a key to look up a modified version in the table.  If one is found, it's used in place of the original, but if no match is found it gets written out to the error file.
 

I ran into a similar situation at work with a couple of pipe-delimited files, but in those cases there was just one free-form field involved.  In those cases you can assemble everything before the offending field, then everything after, so that any pipes in what you have left can get automatically escaped.
 

[tjharrell]

On 5/15/2021 at 11:25 AM, rdunajewski said:

 

If the FCC is able to commit to fixing it, perfect. It'll resolve itself once they fix the weekly database file. If they don't, then we have to find a reliable way to fix this and future cases, which is really suboptimal.

I wouldn't hold my breath.  I've been playing around with these for almost a decade and they've been 'broken' for at least that long.  If it makes you feel better, it's not just ZA.

[Citizen]

4 hours ago, tjharrell said:

I parse these things too, and what I do is write out all the parsing errors to a separate error file for manual review.  Those records go into a two-column database table with the original record as received from the FCC in one column, and a modified parseable record in the second.  Then the next time the record is parsed, if there are any parsing errors then the record is used as a key to look up a modified version in the table.  If one is found, it's used in place of the original, but if no match is found it gets written out to the error file.

I ran into a similar situation at work with a couple of pipe-delimited files, but in those cases there was just one free-form field involved.  In those cases you can assemble everything before the offending field, then everything after, so that any pipes in what you have left can get automatically escaped.

Agree.  And I'd bet the file that Rich is getting from the FCC is one of those that has free-form fields (Rich can correct if wrong).  A simple string scan/change should do the trick, probably don't even need to assemble the other fields (should already be assembled properly).

 

4 hours ago, tjharrell said:

I wouldn't hold my breath.  I've been playing around with these for almost a decade and they've been 'broken' for at least that long.  If it makes you feel better, it's not just ZA.

I would propose a one-time write of a simple script to do the string-change according to what Rich described above.  That same simple script could be run each time he receives the file from the FCC.  And if ever needed, it could easily be modified to accommodate any new string errors that may subsequently appear.

...

[rdunajewski]

I have fixed the issues with the license importing script. I never heard back from the FCC, so I had to put in a hack to pre-process the files looking for a certain case.

WRMM237 and WRMN710 are both in the system now.

[Seejay21]

On 5/21/2021 at 3:43 PM, rdunajewski said:

I have fixed the issues with the license importing script. I never heard back from the FCC, so I had to put in a hack to pre-process the files looking for a certain case.

WRMM237 and WRMN710 are both in the system now.

Thank you!