|
Member 14726719 wrote: Someone trying to parse TBs worth of JSON or trying to shoehorn a small amount of JSON parsing into a very small space in an embedded processor could be very grateful for that extra work!
Which is funny because those are the two exact things I designed this library for. 
Real programmers use butterflies
|
|
|
|
|
Then your effort isn't wasted. You're just not testing it with the right data set / machine configuration... (which would make the difference much more noticeable). For example, throw enough JSON at it that it takes a couple hours to work through it... you'll be really grateful for that 2x perf then. Or find a small embedded processor and download and run it there... (it could be processing configuration data, schedules, diagnostics, sensor data, or programming rules - like a Programmable Load Controller using JSON setups).
|
|
|
|
|
I don't have terabytes of space or an on-the-fly streaming JSON generator to make my PC run for hours over some JSON data. I would need many terabytes for that.
And on the embeddeds I'm using some real world data on the large side, and it takes seconds to parse. I can't really improve it though because it's on a dimestore 8-bit cpu not anything superscalar and it can barely keep up with the SD reader at half speed.
Real programmers use butterflies
|
|
|
|
|
Yep. Real world problems stink... You could always pick up a inexpensive external HDD or Thumb Drive (depending on how much data you need) and load it up with JSON files... you really don't need something that runs for hours, just something long enough that people would be willing to pay and/or update their software for something faster (or smaller).
I used to do code optimization on an 8080, that's was fun - counting the cycles of each instruction... writing my own multiplication routines...
|
|
|
|
|
So far I've been fine with 200MB. Eventually I may try 1GB or more, but I think 200MB of data is plenty for me to profile with. When I run the gnu profiler over my code, the timing is plenty with this dataset.
Real programmers use butterflies
|
|
|
|
|
No code "optimization" giving a twofold speedup (or less) is worth it.
Optimizing a library function consuming ten percent of the total CPU power consumed by a production run of your application (or less) isn't worth it.
First law of code optimization: Don't do it.
Second law of code optimization: If you have to do it, don't do it yet.
"Optimization" is one of the most poorly understood, and therefore one of the most badly abused, concepts of the software business.
|
|
|
|
|
I'm not sure I agree with your thresholds but I at least can respect the sentiment.
I'm satisfied with the optimizations I have done. Most were more than a 2 fold speed up. More like 4x-6x overall speedup in final throughput, with every optimization i made considered together.
But where most of the optimization was was in ram use, and every kB counts when you're building something that runs in under 4kB - because that's all some machine have for RAM. 
Real programmers use butterflies
|
|
|
|
|
back when I was building Industrial control software (some of these systems had over 10,000 I/O points), every processor tick counted for getting as close to real time as possible. When I had to use other's libraries: I graded them on speed first, ease of use second.
one of the bigger reasons for speed was: I had no idea what the final computer would be on specs so my test computers would be at least 10 years old running XP with 2GB ram, and a 800mhz processor. if it ran fine and kept the CPU< 5% for normal load, I was pretty confident that it would run on any PC the customer supplied with out much issue.
the embedded industry is even more hard core, and is fun challenge to work with the least amount of resources possible to not max out the CPU or memory on those little chips.
I still bring that same mentality when working on any platform now. If I'm building a website or a desktop app or service, I have no idea what the end users setup is going to be like and may be trying to run still on an 800mhz because they can't afford anything newer. (and yes I know of a few people in this situation).
so yes working on any project to make it just a little faster is a definitely a  with me, and likely many others out there.
|
|
|
|
|
Well, in my case there's a significant downside to optimizing the way simdjson does - simdjson must index the entire document, and while it does so very quickly, it uses gobs of RAM.
My parser will run reasonably on a 4kB machine. 
Consider that this "twice as fast" scenario is only on workstations and servers that are 64-bit and like i said, requires gobs of RAM.
Even without it I get massive throughput, even on old PCs so what's the point of going further than that when doing so requires me to load the entire document into RAM, and makes it so it can't run on an 8-bit machine - won't even compile.
That was my calculus. I hope that seems reasonable, but if it doesn't I'm more than happy to discuss it. I'm just a nerd like that.
Real programmers use butterflies
|
|
|
|
|
First of the year, it's like a Winter Wonderland out there!
As in "I Wonder when that white cr@p is going to bugger off?" 
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
Saw a documentary about a Belgian couple that emigrated to Norway, they discovered a bit too late that the region they were in was notorious for the amount of snow falling. Only a few months of the year they did not have to shovel snow.
|
|
|
|
|
We had a couple move in next door to us one fall. In the spring they sold the house because they couldn't handle the snow. They were from Wisconsin and I'm in Denver.
|
|
|
|
|
And they didn't enjoy riding a one-horse open sleigh?
|
|
|
|
|
So do you prefer rain and mud?
"The only place where Success comes before Work is in the dictionary." Vidal Sassoon, 1928 - 2012
|
|
|
|
|
I am sure he prefers drought and famine. 
|
|
|
|
|
Well ... it isn't cold and slippery, and it doesn't have to be chipped off the car windows with a pneumatic drill ...
Snow was easier when I rode motorcycles all the time: wipe it off the seat, turn on the heated grips, and ride away - no need to worry about the speedo, you weren't going to be going fast enough to exceed speed limits anyway ...
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
OriginalGriff wrote: it doesn't have to be chipped off the car windows with a pneumatic drill ...
Concentrated washer fluid in a spray bottle. (CalciumChloride is the most efficient solution, but it will corrode your car)
|
|
|
|
|
|
Dry and frosty over here when I got up. But it started snowing about half an hour ago. It looks like it has been gradually moving east.
|
|
|
|
|
Snow started about 10am in Watford, Herts. 2" - 3" by noon. Easing off now (1pm)
|
|
|
|
|
We had a bit more here in Woking, but stopped now, and the sky is actually a bit brighter.
|
|
|
|
|
Looling outside now Richard I think it'll be mostly gone by the morning - I'm only a coulpe of miles away from you.
"I didn't mention the bats - he'd see them soon enough" - Hunter S Thompson - RIP
|
|
|
|
|
More than likely, but it depends whether it freezes tonight. I must have driven past your house on more than one occasion over the past 10 or so years.
|
|
|
|
|
You probably have Richard it's down a side road opposite St Paul's Church in Addlestone
"I didn't mention the bats - he'd see them soon enough" - Hunter S Thompson - RIP
|
|
|
|
|
Yes that was a shortcut I used to go home quite often; but the centre up Rudge Rise closed a couple of years ago, they are now based in Walton-on-Thames.
BTW Pete, the snow is still thick on the ground here.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
