|
If only!
But the data is supplied on file supplied via SFTP - and nothing else is obfuscated - including other percentage figures 
|
|
|
|
|
CHill60 wrote: and decided that the number should be divided by 10 to the power of the number of decimal places defined.
That actually makes sense. With some of the credit card processors I deal with, all amounts are integers with an implied number of decimal places, usually 2, so 1234 is 12.34. So yeah, if I had a spec that said the number of implied decimal places was 4, then 1234 would be .1234
And yes, only in Financial Services. 
What I would be more concerned with is, when they divide by 10^n, what's the data type that's holding the result? Hopefully not a float or a double!
|
|
|
|
|
Marc Clifton wrote: And yes, only in Financial Services.
I disagree...
In the Automation world, this is used pretty often.
I.E.
you get analog inputs i.e. [0, 10](V) transformed into decimal (depending on manufacturer or configurations 0 to 27658 or 0 to 32768)
to save data transferred into de bus (i.E. for the SCADA Visualization) is usual to send the int16 or Word and say in the panel, this variable contains 2 decimals (1234 = 12.34) or 4 decimals (1234 = 0.1234)
This is because the float / double would be 32 bits long, which means you save 2 bytes per variable using this method.
And this is still actual because (one of the reasons) the standard "Profi-Bus" speed is 1.5 mbit/s, saving that much bytes still helps.
M.D.V. 
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
modified 13-Dec-19 9:14am.
|
|
|
|
|
Nelek wrote: In the Automation world, this is used pretty often.
Ah, very true -- I remember doing exactly that sort of thing in the good 'ol days of assembly language programming and interfacing to a variety of hardware.
|
|
|
|
|
PLC LAD is kind of similar to assembler
And still in use almost overall.
M.D.V.
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
|
I'm learning stuff today!
|
|
|
|
|
About this concrete case, I can't say much.
But I have used it a lot in automation. For more details see my answer to Marc.
M.D.V.
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
|
This is a really old programming problem. How would you store a running total of a bank account? Since it has dollars and cents, how about a float?
Ok, so then you run into the "floating-point rounding error" problem.
>>> 0.1+0.1+0.1
0.30000000000000004
Well... that's pretty bad to be adding/dropping pennies every trillion transactions or so because of cumulative binary representation of decimal digit errors. What else can we do?
One of the commonly chosen ways is to used fixed decimal. i.e. Assume a fixed number of decimals and represent everything as integers. One no longer has to worry about fractions of a penny creeping up on you over time if you work in discrete units of pennies. i.e. 100-97 = 3, never 0.30000000000000004
Well... then your manager comes in and says, we need to be able to track hundredths of a penny because the state tax paid even including fractions of a penny. So it is deemed that 4-digits is enough precision to satisfy the need. Now 10000 - 9700 = 300 is still okay, is never gonna be 300.00000000000004 and the same situation applies as before.
But your front-end buddies didn't update the website to account for the fact that the database now assumes 4 digits of precision. And thus the problem is born.
Fun trivia: "Office Space" movie characters took advantage of this to siphon off millions of dollars back when this was a thing in the 80s-90s-00s.
|
|
|
|
|
Yeah - I get that it makes sense. It also saves space in automation etc. TBH we used to do it back in the day (but that was sooo long ago I'd forgotten).
My main gripe with this was that this is the only feed (of many) that was defined like this, and it's not the case for all the fields  It looks like the whole shebang was designed by committee!
To make things worse the BA involved shares specifications a page at a time, only the bits that she thinks are necessary to do a job (that she is not qualified to do nor has any experience of). This was not on the pages she chose to share with me 
|
|
|
|
|
I have an app that runs on multiple (Windows, iOS, Android - as both a native app or as a web app).
JSON is a nice lingua franca -- I thought.
You see my app has simple string-based keys the user adds to keep track of her sites.
Since I'm not sure what the user might type, I go ahead and encode the keys to Base64.
Normally you see some Base64 encoded keys which look like:
c3VwZXJzaXRl
c2Vjb25kU2l0ZQ==
dGhyZWU=
Serialize Object As JSON
In my web apps and on Windows I serialize all the user's sites as objects via JSON and it works great and looks like:
[{"HasSpecialChars":false,"HasUpperCase":false,"Key":"c3VwZXJzaXRl","MaxLength":0},{"HasSpecialChars":false,"HasUpperCase":false,"Key":"dGhyZWU=","MaxLength":0},{"HasSpecialChars":false,"HasUpperCase":false,"Key":"c2Vjb25kU2l0ZQ==","MaxLength":0},{"HasSpecialChars":false,"HasUpperCase":false,"Key":"eWV0QW5vdGhlcg==","MaxLength":0}]
Of course on web sites (JavaScript), its the old JSON.stringify(allSiteKeys) that handles that so nicely.
And on Windows I've always used NewtonSoft libraries and it all works great. All interchangeable.
Android Gson - Google's JSON
Enter the problem.
However, while developing on Android I wanted to serialize the data the same way so I turned to Gson which is Google's official Android way of doing this.
However, I noticed that values which were Base64 encoded properly were output to JSON in an interesting way. Every equal sign was altered to \u003d which is the unicode equals sign[^].
That means my JSON would be altered to look like:
[{"HasSpecialChars":false,"HasUpperCase":false,"Key":"c3VwZXJzaXRl","MaxLength":0},{"HasSpecialChars":false,"HasUpperCase":false,"Key":"dGhyZWU\u003d","MaxLength":0},{"HasSpecialChars":false,"HasUpperCase":false,"Key":"c2Vjb25kU2l0ZQ\u003d\u003d","MaxLength":0},{"HasSpecialChars":false,"HasUpperCase":false,"Key":"eWV0QW5vdGhlcg\u003d\u003d","MaxLength":0}]
NOTE: This is not a case of oddly encoded Base64 -- if I look directly at the Base64 encoded data it still has the equals signs. This only occurred when the data was serialized to JSON.
Searching For An Answer
It wasn't easy to find the answer, because a Google developer explains it this way[^]:
Google dev = is a special javascript character that must be escaped to unicode in
JSON so that a string literal can be embedded in XHTML without further
escaping.
XHTML? Oh, are we back in the year 2000?
It made no sense to me and others who were like, "Uh, JSON is a format for transmission. Why would I care if there is an equals sign in the data? It's just bytes."
It is quite difficult to find docs on the Gson library. Not great stuff.
But, finally, I found a stackoverflow answer[^] that mentions that you have to use the Gson builder to create the initial Gson object, and when you do you have to turn off html escaping...
Gson gson = new GsonBuilder().disableHtmlEscaping().create();
Normally, that would just looke like:
Gson gson = new Gson();
Everything else is the same so when you serialize it you still call the same code:
String jsonSiteKeys = gson.toJson(allSiteKeys);
The GsonBuilder() is almost like magic, because how would you ever know it is there?
Different Dialect Which Includes HTML Escaping
The final point is "Why would the dev think that the normal case is to include HTML escaping when this is a transmit format?" Why not think that is the special case? Especially since other libraries handle it without the escaping.
All That For Compatible JSON?
That is a sick amount of time just to get JSON in a compatible format.
In the future when AI takes over all development work, how will it handle this? It will not allow anyone else to write JSON libraries.
modified 9-Dec-19 17:03pm.
|
|
|
|
|
|
Thanks for the link, I will check it out.
|
|
|
|
|
|
|
I once had a similar issue with a GUID.
Apparently there are different ways to look at a GUID 
I'm not really sure if it was anymore, some encoding issue or whatever, but two applications showed the same GUID differently 
|
|
|
|
|
Sander Rossel wrote: I'm not really sure if it was anymore, some encoding issue or whatever, but two applications showed the same GUID differently
If you look at Create GUID (Visual Studio Tools) - snapshot[^] you can generate GUIDs like:
All the same value, but different formats.
1. <guid("0c2d39cd-f486-435b-bdba-64a124d04d31")>
2. [Guid("0C2D39CD-F486-435B-BDBA-64A124D04D31")]
3. {0C2D39CD-F486-435B-BDBA-64A124D04D31}
4. // {0C2D39CD-F486-435B-BDBA-64A124D04D31}
static const GUID <<name>> =
{ 0xc2d39cd, 0xf486, 0x435b, { 0xbd, 0xba, 0x64, 0xa1, 0x24, 0xd0, 0x4d, 0x31 } };
5. // {0C2D39CD-F486-435B-BDBA-64A124D04D31}
DEFINE_GUID(<<name>>,
0xc2d// {0C2D39CD-F486-435B-BDBA-64A124D04D31}
6. IMPLEMENT_OLECREATE(<<class>>, <<external_name>>,
0xc2d39cd, 0xf486, 0x435b, 0xbd, 0xba, 0x64, 0xa1, 0x24, 0xd0, 0x4d, 0x31);
39cd, 0xf486, 0x435b, 0xbd, 0xba, 0x64, 0xa1, 0x24, 0xd0, 0x4d, 0x31);
|
|
|
|
|
Yeah, but this was really like one tool showing "abc" while another showed "123" and they still somehow had the same underlying value
I remember this was a thing on MongoDB, so I just installed Robo 3T just to look at the options and it's Legacy UUID encoding (do not decode or use Java/.NET/Python encoding).
A coworker used another tool with another encoding and we were looking at the same document, but different values 
|
|
|
|
|
Sander Rossel wrote: Yeah, but this was really like one tool showing "abc" while another showed "123"
That is interesting and painful. Very similar to my problem.
|
|
|
|
|
Mostly painful 
|
|
|
|
|
Oh no my friend, it gets far worse than that  A GUID is simply an 128 bit number in 5 chunks of 32-16-16-16-48. GUIDs like {0C2D39CD-F486-435B-BDBA-64A124D04D31} are when the first chunk of 32 bits is converted into hex, then the next chunk and so on, but when you use some dumb-ass third-party system that stores the bytes in a different order *cough*mongodb*cough* then the string representation of the same data can look different in two different systems despite being equal. I know this is more of a UUID issue than a GUID one, but it shows the kind of nightmares you come across when you stray from the Microsoft path.
|
|
|
|
|
Quote: when you stray from the Microsoft path
Stay on the path! Mirkwoodicrosoft has many hidden dangers!
- I would love to change the world, but they won’t give me the source code.
|
|
|
|
|
+1 for the {4C 4F 54 52} reference
|
|
|
|
|
F-ES Sitecore wrote: A GUID is simply an 128 bit number in 5 chunks of 32-16-16-16-48. GUIDs like {0C2D39CD-F486-435B-BDBA-64A124D04D31} are when the first chunk of 32 bits is converted into hex, then the next chunk and so on, but when you use some dumb-ass third-party system that stores the bytes in a different order *cough*mongodb*cough* then the string representation of the same data can look different in two different systems despite being equal.
Oh that is really terrible. You'd think they'd mention Big-Endian v. Little-Endian or some such.
|
|
|
|
|
Sander Rossel wrote: different ways to look at a GUID
Yes, there are "case-sensitive" Guids. I guess it was project ids in a Visual Studio solution file...
Oh sanctissimi Wilhelmus, Theodorus, et Fredericus!
|
|
|
|
|
A good reason to ALWAYS add your own abstraction layer on top of ANY API that you use.
It is a lot easier to do it up front then try to retrofit it later.
This keeps your dependencies well documented as well.
String jsonSiteKeys = MyLib.json.toJson(allSiteKeys);
Or maybe better
String jsonSiteKeys = MyLib.toJson(allSiteKeys);
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
