|
|
As Daniel Lemire initially suggested, I started a pull request.
added jsoncpp folder for comparison and possible adaptation/integration by codewitch-honey-crisis · Pull Request #1367 · simdjson/simdjson · GitHub[^]
I did this in order to continue the discussion. That way others could access my code. I haven't integrated in that request, because our apis are so different, and mine needs a feature add before I start anyway.
Anyway, it's all under the jsoncpp folder, including the very beginnings of a comparison benchmark in main.cpp under that folder, and a small build/run script for linux as ./run.sh
I'll add more comparisons once i figure out how to use ondemand to search across each different axis (forward, descendants, siblings) - right now i can only get it to do siblings, but it's not fair to make it walk the tree in the code if it doesn't have to, when mine doesn't. Nor do i want to write skewed benchmarks just because i am still learning the api.
Real programmers use butterflies
|
|
|
|
|
yeah,
It will be interesting to see how your lib performs against the others. I added a reminder in my calendar to check up on it next week.
|
|
|
|
|
Eventually I found out that there's an odd compiler switch "-march=native" that makes simdjson scream.
Once I enabled that, it was basically like simdjson vs rapidjson.
I had everyone going there for a little while though, including myself.
Nobody caught the compiler settings until way late in the game. I've never used that switch myself.
I'm still working with simdjson because even after all that, my code is still apparently very fast when compared to their competitors, according to them anyway.
And they still want streaming support, my code is apparently interesting enough that they haven't kicked me off their github yet.
I'm still happy with the results. I haven't spent a year optimizing my code and I'm keeping up with their competitors.
And I can still make it faster. 
Real programmers use butterflies
|
|
|
|
|
The -march=native option tells GCC that you want to optimize for the machine you are compiling it on. I never even thought to question your compiler/linker options. I just assumed that you were experienced enough to have that perfected. Come to think of it... I don't even remember seeing your Makefile let me look to see if I have it.
Update:
Looks like you were using a Makefile loaded from your environment. You never distributed it.
|
|
|
|
|
Actually I was just using a small shell script that just ran the compiler, and then the program.
I posted it on the site along with my benchmarks.
I'm used to msvc, still newish to gcc and g++.
I thought -m64 covered it for my machine.
Real programmers use butterflies
|
|
|
|
|
honey the codewitch wrote: I thought -m64 covered it for my machine.
No, that would mean it was generating x86-64 code from circa 1999. In fact you were probably running simdjson without any of the modern simd instructions.
Don't beat yourself up, your lib has great benchmark numbers.
|
|
|
|
|
Yeah I eventually found that out. My benchmarks are now where i expected them to begin with, competitive with most offerings. It was when i started getting those numbers against simdjson that i was giddy.
my first goal was ram use, because extreme portability. my third goal after those two was speed.
what i'd like to do is figure out how to get simdjson's stage 1 analysis to stream like my lib does, even at the cost of some branch misprediction.
but before when I thought i was neck and neck i thought simdjson's stage 1 was the wrong answer for bulkloading. it still might be, if i can't stream it and get good performance.
Real programmers use butterflies
|
|
|
|
|
Good that I have none at the moment. In the last two weeks I made some progress with the Zwölf and now all bits and pieces are falling into place.
I wanted to boot the Zwölf with a microcontroller, but was not able to get one anywhere. Now what?
Why not take the simple breadboard Zwölf and add an IDE controller? Then I only need some interface and I get a smart disk controller.
What kind of interface? Serial? Parallel? Both would be kindof slow. I need something faster. Something that can be used for bootloading as well as at runtime. DMA in both directions, that's it.
The CDP1802 never gives up control of its bus, not even for DMA. It keeps running and inserts a DMA bus cycle when DMA is requested. It even acts as a DMA controller and does the memory addressing itself, leaving only putting its data on the bus at the right time to the requesting device. Super easy, only three cheap logic ICs on each processor board. Even a 'you've got mail' interrupt is included to be used when all bytes of a request have been sent. The coolest part is that the processor can even do this kind of DMA when it's not running. The interface can be used to load code into memory before booting. That's going to be a smart disk controller.
Coming to think of it, why not even add some sort of network adapter. A smart disk and network adapter. Let's boot the Zwölf from the internet.
The software side is also coming along. I'm going to be stuck with bit banged software RS232 for a while, so I have written a little library for VT100 terminals and pimped the software from 9600 baud to 19200 baud. No comparison to the VT52 and 300 baud we had in the old days. Just fill the screen when you wanted to take a break.
And I now have a heap. Yes, that's memory management. That may be an alien and revolutionary concept, but there is no hand waving when you start with an empty computer. I even had to use something like a global variable, because it's hard to allocate memory on the heap when the computer does not remember where it put the heap.
I have lived with several Zen masters - all of them were cats.
His last invention was an evil Lasagna. It didn't kill anyone, and it actually tasted pretty good.
|
|
|
|
|
What MCU are you looking for? I can't imagine nothing being available.
I'm not sure how many cookies it makes to be happy, but so far it's not 27.
JaxCoder.com
|
|
|
|
|
I would have to dig through countless datasheets, but I think it was one of the PIC16F18000 series.
I have lived with several Zen masters - all of them were cats.
His last invention was an evil Lasagna. It didn't kill anyone, and it actually tasted pretty good.
|
|
|
|
|
I'm looking at Mouser's site, at a through-hole PIC16F18325-E/JQ with 14k of flash and 1k of ram.
They're listed at $1.62 ea, packed in a tube. (and not delivered)
A few months ago, I bought 5 BluePill boards. They've got an STM32F103C8T6 onboard that's got 64k of flash, 20k of ram and run at 72mhz. They turned up on my door for $13.66
They're a 32 bit Arm Cortex-M3. Wish I'd bought em years ago.
2/5/10PCS STM32F103C8T6 ARM 32 Cortex-M3 Minimum System Development Board Module | eBay
EDIT: Honey has been using some ESP32 boards. They're cheap, faaast and have large memories. Built-in wireless too, if that's important. You're talking about 600 mhz, a meg of memory and still under half a dozen dollars. Can be prone to overheating.
modified 3-Jan-21 22:42pm.
|
|
|
|
|
The ESP32s have 520kB of ram, some reserved, but expandable via SPI PSRAM.
They run at CPU clock of 160Mhz 240Mhz, but SPI clock locked to 40Mhz for the external, although it's possible to get a bit faster than that with 4-bit SD readers in SD MMC mode because the ESP32 is aware of them. SPI RAM also has special provisions to make it faster (80Mhz I think?) That's my biggest bottleneck there - SD and display over SPI. It makes me want to get out and push. JSON(C++) runs at about 74kBps on my little esp32. i *might* be able to speed it up with buffered reads, but it didn't help on by desktop so i took that feature out, so it's possible that you might get better results with fread() than fgetc() - my pc didn't (surprising, i know). Anyway, SPI i/o is not great, so it's a chip to avoid if you want to do say, full speed writes on microSD cards.
They're immensely connectable - bluetooth, wifi, OTA (though bluetooth libs and OTA are each huge so you often have to pick one), and ESP-NOW radio comms. You really have nothing you can't interface with.
The arduino framework is somewhat limiting though. Once I broke out of it and started using FreeRTOS and the espidf directly it opened some doors. multithreading and taking advantage of the second core for example. it's possible to do under the arduino framework but none of the libs are thread safe so why bother?
I think I want to move to the arm platform for my next IoT widgets.
I will probably always love these little chips though.
Real programmers use butterflies
modified 4-Jan-21 10:04am.
|
|
|
|
|
|
I was thinking about getting some of those. I just don't have a use for it yet. 
Oh and I said 160Mhz but I should have said 240Mhz. For some reason 160Mhz came up on the ESP forums and now I can't remember why but it stuck in my head.
Real programmers use butterflies
|
|
|
|
|
I don't s'pose you've tried one of the ESP32Cam packages around have you?
I've been thinking about slapping one into some flying r/c models and am curious about any gotchas.
|
|
|
|
|
Not yet. From what I've seen on the forums they can be difficult. Mostly getting the ESP32 to keep up if you're trying to store screens of the video onto SD and things like that.
But I suppose it depends on what you need to do with it.
Real programmers use butterflies
|
|
|
|
|
Oh, you're a champion. Thankyou so much.
Perhaps I'll just spend more. Funny that people tightarse on something like this, then eat takeout or in a restaurant. We can be such dummies.
|
|
|
|
|
Yeah that might be best. I'm not entirely sure how suited the ESP32 is to cam applications. Don't get me wrong, I know people have done it - I've seen it on reddit's esp32 forums, but like I said, it strikes me as difficult to get the ESP32 to keep up, and I think that's because of its SPI bus limitations.
Edit: Speaking of spending more, be careful what ESP32 devboard you get. I've had bad luck with Adafruit stuff despite it being pricier. I've had good luck with hiletgo's offerings, despite being "chinese junk" and so i've bought plenty of equipment from them. I also like the DOIT devkit board.
If you can find a WROVER model as opposed to the more common WROOM model it will net you 4 additional megabytes of flash though, for a total of 8MB
Real programmers use butterflies
|
|
|
|
|
Ooooh. That's another one against the mighty fruit machine, thanks. I'll keep avoiding them. Half the reason I'm playing with these things, is that they're dirt-cheap and ubiquitous.
To a degree, I actually get-off on buying the cheapest, nastiest chinese junk and wondering if it'll work while I await its delivery. It's a thrill that reminds me of childhood. I'm about to solder together a GSM modem module and see if I can send myself a text-message. It was $2.38 usd on my door! (And I've since read how it's a bit nasty!!  )
WROVER vs WROOM flash size - goodie, thank-you.
|
|
|
|
|
I recommend some hiletgo gear then. The devboards are like $6. Also they sell things like 5 packs of SD-reader modules. Got them for $7 on amazon. Probably cheaper on bangood but i don't like waiting for their shipping.
I also found a 5 pack of some ESP32 nodeMCU clones - i don't remember the brand, for like $25 on amazon.
ESP32 chips themselves (not the devboard) usually come with 4MB. The 8MB and 16MB ones are custom order so they are more rare because the devboard manufacturers simply don't custom order those chips
I've only ever seen one 16MB dev board and it was in an arduino form factor  i don't remember where i saw it. I also don't remember what espressif (the people that make the ESP32 chips themselves) call the 16MB version.
Good luck finding one.
I'd go with the WROVER though. 8MB is plenty, and easier to track down. WROOM is even good with 4MB. I haven't really needed more.
Real programmers use butterflies
modified 4-Jan-21 10:07am.
|
|
|
|
|
Thanks mate, you're worth your weight in Palladium. 
|
|
|
|
|
You're quite welcome. I love those little things, and have been tinkering with them (now professionally too) for a little while, so i've learned a couple things about a couple things where they're concerned.
I'm always happy to share what I know.
Real programmers use butterflies
|
|
|
|
|
I bought a couple of the esp with camera a while back but haven't been able to play with them. My I want to do list is getting so long that my stickie app is running out of room on my monitor...time to get a bigger monitor?, ill put it on the list...if there's room.
I'm not sure how many cookies it makes to be happy, but so far it's not 27.
JaxCoder.com
|
|
|
|
|
Quote: I wanted to boot the Zwölf with a microcontroller, but was not able to get one anywhere.
Why?
"In testa che avete, Signor di Ceprano?"
-- Rigoletto
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
