iguana is a full bitcoin node. one of the things that makes it special is that I have abstracted out all the differences between the various bitcoin compatibles. Well, at least I tried to (please ignore all the special case code for offbeat coin differences)
Most clone coins are basically bitcoin with a few parameters changed, so by putting these changes into a chain parameters structure, most operations for most coins can be done using the same code. Even the same executable!
What that means is that you can run one iguana and keep telling it to become a peer for coin after coin.
While this sounds pretty cool, it creates a testing nightmare. Networking is error prone. errors just happen especially to random peers, so the total possible states the iguana can be in is quite large and while I have made it so it handles most of them properly, it is still possible to get it to crash, especially if more than one coin is run at the same time.
The reason is that the same internal resources are shared among the different coins, so just the scheduling of requests ends up contending with each other and using some randomized backoff algo prevents horrible logjams, it certainly doesnt create any predictable behavior. With 6 coins at once, even if there is a small chance of crash in isolation, the combined crowd of coins makes it so that iguana is crashing way to often to be ignored.
So, into the debugging I went.
Immediately I ran out screaming.
Oh, no, that was in a movie. Sorry.
But I did get this bad feeling like this would be a never ending search for all possible edge cases that combine with other edge cases. It could easily burn a day or three without any guarantee of a happy ending.
[Foreshadowing: 9 coins all parallel syncing at the same time without any crashes]
I like to visualize what I want to achieve to help myself figure out how to solve difficult problems. Sometimes I just get messages from the future, but cant always rely on that, especially when there are a lot of sunspots.
I reviewed the code and found some places that violated the strict safety guidelines, fixed those. Found a few more, fixed those, but still crashes with 3+ coins... Then I took a nap. If I dont have a solution in mind, then it is usually a waste of time to keep pounding away at it, plus I think it was already past midnight anyway.
A few hours later I wake up with the answer! It was in front of me all along. Things are quite stable with just one coin at a time. So building on that I added a command line parameter which if it is a file with valid JSON in it, I just run it after initialization is complete. iguana codesize is all of 2MB, smaller than a high res photo. so running many iguana's wont cause the system any issues (ignore iguana's ability to parallel sync thousands of bundles from thousands of peers at once) as long as the settings of all the instances add up to a manageable load.
so, like real iguanas that seem to like to congregate in large groups, my iguana will do the same. During writing this post I have been syncing 9 coins at the same time, using 9 different iguana processes. Now even if any one of them crashes, the others arent affected and I can even make an infinite loop shell script to just restart any crashed iguana, so it would look like the iguana is always busy and well behaved.
James
The physical memory used is the RES minus SHR, so even though it looks like a lot of memory is being used, most of it is in the form of memory mapped files, which is like VM for the process, but I use only a read only maps to be compatible with all OS. So by and large each coin is using about 1GB or less, the variance is how many bundles in parallel I have each one doing.
(if anybody knows how to make it a mono-spaced font, help is much appreciated) <- thanks 
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19058 root 20 0 34.342g 5.847g 4.765g R 116.1 18.7 15:38.99 iguana
18785 root 20 0 0.122t 2.415g 2.135g R 97.4 7.7 16:32.60 iguana
19494 root 20 0 4815292 2.638g 94132 S 96.8 8.4 17:25.94 iguana
19397 root 20 0 7540568 1.553g 233528 S 84.8 5.0 15:32.04 iguana
18753 root 20 0 4272296 1.084g 466300 R 81.5 3.5 11:38.63 iguana
19203 root 20 0 12.285g 3.903g 2.049g S 76.2 12.5 15:38.13 iguana
19536 root 20 0 1778864 391844 178952 R 62.5 1.2 13:27.82 iguana
19348 root 20 0 16.187g 3.264g 2.317g R 62.2 10.4 14:13.39 iguana