Upgrading and maintaining a home RAID installation is kind of a bitch — just saying. Especially if you’re cheap like me, and you have a Franken-RAID.
My current server has 4 striped sets of paired drives — none of the pairs is the same size as any of the other — my server had 8 bays, just I would just buy two drives in pair when I needed to upgrade — a new controller card if needed — create them as a new striped set — boot into windows — and expand the volume across the new set.
That worked great for a while — but Windows doesn’t allow you to span a volume across external drives (if the removable bit is set to true, the options are grayed out — if you do it while they’re internal, and then move them to external, it just pretends the drives are unformatted and shows them as RAW, it’s super silly…) — and also buying them in pairs is a bit expensive sometimes ($200 to add instead of $100…) — and… ZERO MOTHER-FUCKING REDUNDANCY… I mean… it’s worked great for like 6+ years now… and I’ve got 7 TB of space… but every now and then, I worry a little — you know? I mean crap, re-downloading all of that porn will take FOREVER.
So, RAID-5, right? — but then all the drives have to be matched size… also, I have no experience with RAID 5 — can you just add another drive to the mix willy-nilly and it works? — or do you have to reinitialize the whole array? (Can you reinitialize it after you add the drive, and not loose all your data? Is there a way to do this? And then… you’re limited by the number of ports on a single RAID card, no? So every time other time you upgrade you need a new card, that sounds like it would get very-expensive, very quickly…)
So, let’s take a step back — shouldn’t somebody have solved this problem by now? — Maybe a Linux soft-RAID driver that just lets you throw whatever you want at it, and it figures out the best, and most redundant config for your drives, and makes everything just work? And tells you when one failed and then you just go replace it, and everything’s happy? Am I dreaming?
Before the nay-sayers start claiming it can’t work — The problem — from an algorithm POV — doesn’t seem that hard to solve; here, I’ll walk you through how one implementation could possibly work (it was just right off my head, so it’s probably not even the most efficient):
For nostalgic purposes — and also just for the heck of working with smaller, more text-book-like, numbers — let’s say we’re building a RAID with 60 GB, and 80 GB drives (it’ll be fun!).
Let’s ignore the R part of RAID — just for a moment (we’ll factor it back in at the end — but for the sake of not putting the cart before the horse, let’s factor it out for now):
Each of these drives — 60 GB, and 80 GB — have many sizes that they can divide by evenly — for the sake of simplicity let’s say 20 GB for now (in a real implementation, you would use a smaller size, between 512 MB, and 2 GB, I think — just because, the drives will never match up so evenly — but let’s say they do — for simplicity sake, and that they can all be divided evenly by 20 GB…).
That means, we can treat the 60 GB drives like 3 LOGICAL 20 GB drives, and the 80 GB drives like 4 of them, etc, etc. — So — we can stripe those together — without parity (because it wouldn’t make any sense, right?) — so far, not a lot gained over a spanned configuration — sometimes, you might get an IO speed boost — but not always — and the processor hit would kind of suck — but we did it — we combined the drives together for their total size — albeit in a way that was kind of dumb — but it allows us to factor redundancy back in quite easily now…
So how do we do that? — Well, stick with me for a moment, before shaking your head — but you’ve heard of RAID 5, yes? — It uses N+1 drives of equal size — and you can get the full storage space of N (the storage distributed across the N+1 drives evenly, along with a parity that is also distributed across the drives evenly). — If one drive fails — it can be reconstructed.
But that’s no good for us, because if one of the PHYSICAL 60 GB drives fails, we lose 3 LOGICAL drives! and god forbid the 80 GB drive would fail — that would be like losing 4 LOGICAL drives!
*dun-duh-da-dun!* Enter RAID 6. It requires N+2 matched drives (and makes N drives of space usable with 2x the distributed parity) — allowing up to two drives that can fail, and still be rebuilt…
So, what about N+3, and N+4? Yep, 3 and 4 drives can fail… Nice, right? (I mean, it’s not commonly done — and I don’t even know if there are implementations that support this — but I see no reason why it couldn’t be done — do you?).
So, let’s say we have a RAID-X setup of 3 PHYSICAL drives — two 60 GBs, and one 80 GB. — let’s split that into 20 GB chunks and pretend it’s really ten 20 GB LOGICAL drives (60 + 60 + 80 = 200. 200 / 20 = 10).
To be fault tolerant for failures of the 60 GB drives you would need a RAID 6 setup of at least N+3 — to be fault tolerant for the 80 GB you would need a setup of N+4 — (so 60 or 80 GB of the whole, respectively, depending on how much you felt like gambling) — so with three drives, you would have 120 (or 140 if you wanted to gamble) of fault-tolerant storage.
I know what you’re thinking — for true fault tolerance, the N+4 setup — we see only 120 GB of space — and that’s the same for if we did a regular RAID-5 and just used three 60 GB drives. — That’s true. But don’t forget about scaling. Let’s say that we add another 80 GB drive into the mix now (because it’s a year later and we want to upgrade a little — and bigger drives are cheaper, right?) — So that makes: two 60 GBs and two 80 GBs — well — that 80 GB just goes right in — and we’re at 200 GB now (or if we didn’t expect the 80s to fail, and wanted to gamble a little, we could squeeze out 220 GB of storage from the array…) — versus with regular RAID 6 you would be looking at those like 60 GB drives (giving you only 180 GB of free storage) — and pretty soon you start adding in 100s — well now you need to be ready to support losing 5 drives (etc…) — but once that second 100 GB goes in, you start seeing the benefits…
Basically you lose the amount of storage as the biggest drive that you want to be safe against — if you only have one drive of that size, it might as well be the next smaller size down, as far as how much good it does you for storage — however — once you add another drive in that’s the same size (or bigger) — you get that space back. — You could also do other things, as aforementioned, you could gamble and only be tolerant for the 60 GB. Or say you had a 10 drive setup — five 60s, and five 80s — you could be tolerant against two drives failing — you could be decide to be tolerant for 80 GBs for the first drive (which would handle any 60, or 80 GB drive failing) — and 60 GB for the second drive (so if two drives fail, and at least one of them was a 60 GB drive, you’re all-set, but only if they were both 80s would you actually lose everything).
There’s probably a way to be even more efficient, and redundant as far a theoretical approaches go, but this was just the simplest, off-the-top-of-my-head solution.
It seems like you could probably do the equivalent of this, manually, with existing Linux soft-RAID Drivers — but every time you added a bigger drive, you’d have to reinitialize the whole array, and loose all your files (because it wouldn’t even be the same level of RAID anymore — you would go from N+3 to N+4, or N+5, etc…, or the evenly divisible amount might change slightly, so the whole stripe nature would change) — also you’d have to figure out every upgrade by hand — backup all your data, rebuild — do it all again — and that would just be a silly waste of time to do all that by hand every time — not to mention all of the backup medium you’d need…
So, is there something that can manage all of the above automatically — and handle the upgrading — reinitialization — repairing, etc. of such? — Does this already exist? — Or do I have to go learn how to write drivers / modules for Linux now?
Also — is there an exiting way to do background drive health checks (maybe something halfway between SMART and CHKDSK full surface scan) without taking the whole drive array down? (I mean, because, there should be — assuming none of the drives are in the DEAD state, you could theoretically operate without at least one whole drive for most setups — so maybe the system could always be systematically testing itself — never one drive all at once — but maybe one or more logical 512 MB block at a time (one for each drive that’s not currently IO bound) — and based on some schedule — you would know how often the full set of drives was checked — i.e. if a full surface scan takes 80 days, you would know that within ~120 days (of moderate usage) the whole set had the equivalent of a surface scan performed on it… no?)
So, last week there was a big hubub when @Josefprusa tweeted:
So I called @makerbot support and asked. Replicator 2 is NOT Open Source! #FAIL #reprap 😦 I’m sad! /cc @bjepson @adafruit @make @hackaday.
Then a few days later, by happenstance, I was browsing SparkFun, and noticed they had a link to the Open Source hardware Definition (it sounds awesome by the way). However, as I was reading it, one thing stood out to me like a sore thumb:
8. No Discrimination Against Fields of Endeavor
The license must not restrict anyone from making use of the work (including manufactured hardware) in a specific field of endeavor. For example, it must not restrict the hardware from being used in a business, or from being used in nuclear research.
Now, I know it’s not in the RepRap license, but it is a statement made on BEHALF OF the RepRap core development team (on a sub-domain of the official RepRap website, no less):
… the RepRap researchers will work actively to inhibit and to subvert the use of RepRap for weapons production, whether by individuals, companies, or governments. And we will remove any such designs from this site. At least RepRap is not very suited to weapons manufacture – it tends to work with more subtle and delicate materials. Give people an internal combustion engine and a few will make tanks; but many more will make ambulances.
Whether you personally support* weapons or not is of no value to this discussion; the topic here is that while the RepRap license may not explicitly state you can’t make them, the core RepRap research team has committed themselves to behaving in a way that is completely contrary to the spirit of the open hardware movement.
While I do not classify them as the same level of evil as I do MakerBot Industries at the moment, I still do not hold the RepRap research team guiltless for their hypocritical and unacceptable behavior.
(*: for the curious, I do support gun ownership, after all: “if you criminalize guns, only the criminals will have them”. — not to mention, as for the Tanks and Ambulances statement, if we’d sent ambulances to Europe instead of tanks, I’m sure World War II would have gone much differently… Just sayin’.
And fuck, guys, think about the children — yes I’m being serious — if you give the community the opportunity to build a safer gun, it may save lives.)
- Update (Sept. 27, 2012):
@FOSSwiki @BrainSlugs83 we haven’t discussed that in core dev, its not official statement. Sebastien put it in there. We will look at it THX
It may not be an official statement, and I as I said before, even if it is, it technically doesn’t violate the Open Source Hardware rules (as their RepRap license does not prevent you from making weapons), but it is still a display of anti-open-source-hardware behavior, supposedly on behalf of the RapRap development team, and is certainly contrary to the spirit of the open source hardware movement, in general.
In the end, everybody makes mistakes, the real measure here will be to see how quickly (or even if) this mistake is remedied. I will definitely be waiting to see what the result of this discussion is.
Hopefully, they’ll have a discussion, come to an agreement, and take the statement down as an “official” action (as opposed to, by some vandal editing their wiki). That kind of remedy would definitely be acceptable, not to mention a testament to the coolness of the RepRap guys. Let’s see what happens. I’ll keep you guys posted.
Apologies, in not updating this thing in forever. I know I may be on a couple of people’s RSS feeds who probably forgot I was in there. 😉
Secondly, I’m not sure if I want to keep this blog going in its current form or not, because I really don’t spend that much time doing game programming anymore and that’s kind of what this blog is supposed to be about. Maybe if things weren’t so fucked up, that’s what I’d be doing…
I love video games though, they’re why I learned to program so young – this year I took part in the Allegro Speedhack, but it will probably be my last (I know I say that every time, but seriously, I’m mortal, I have a limited number of years on this earth, and writing crappy broken code in dead programming languages utilizing horrible tools is not how I envision myself spending that time.) I also had most of my weekend tied up with other obligations… needless to say I didn’t finish a complete game, but it was something I was still moderately proud to have been a part of. Though, participating certainly showed a lack of judgment on my part.
Most recently, I’ve become a lot more politically aware and active. I’ve joined a group of people and helped to found (and modestly grow) the Washington Pirate Party (apologies, we all work and have very little time, the site isn’t that mature yet. Our Facebook page and group is where most of the action is, but we’re hoping to transition away from that.
For those unaware or who think the Pirate Party is about making Napster legal or something – it certainly has some roots in that regard with pissed of teens going “what the fuck is the artificial scarcity shit?” and “why is copyright so out of whack?” – but it’s evolved much past that, the Pirates realize that most of this stuff is a symptom of a much bigger problem that exists in our power structures today.
Fixing copyright is still a core issue, but more importantly are issues of civil rights – including digital civil rights; you deserve a phone that can’t be used to spy on you for no good reason, and lots of bad ones.
Keep in mind also, the importance of the internet (more on this in a moment) the Pirate Party wants a free and open internet for all. Yet in the US congress many bills have been pushed in recent years, with at least 5 bills in the last year: COICA/Protect IP, Internet ID, ACTA, the Kill Switch, and so-called “Child Pornography” bills to name a few – the media tries to portray this as we’re over reacting to the great firewall our government wants on the internet – COICA was literally “Hey, look what China can do! Awesome! Let’s get rid of all this due process and just censor the fucking internet!” … but seriously, they’re still at it. As far as not having a great-firewall-of-china (and several other wins), you have but one senator to thank for stepping up (Ron Mother-fucking Wyden, bitches.) A couple others seem to have come on board now too, including Al Franken, and Maria Cantwell (yes, most bills are named for the exact opposite of what they stand for, or hope to accomplish, not this one, I’ve read it myself – even edge cases/loop holes I thought up were addressed. Not like the official net neutrality bill “with loopholes big enough to drive a truck through.”)
But back to point, the Pirate Party is at its core, there to pirate the government back, into the hands of the people. We are more technology aware than other parties—and that’s a good thing – because when we fight for government transparency and accountability, individual privacy, and your civil liberties – guess where the main fight really is? The internet, that’s where.
We approve of a consensus based and enfranchising approach – we think everyone should have access to political discourse, and that with today’s technology, that everyone can have a voice and be heard. That the person you elect is YOUR public servant and answerable to you, not whichever corporation has the highest paid lobbyists.
So where do we stand on the other issues? Partisanship, etc? Well, for the most part, we don’t. Each candidate may have their own personal views in addition to the Pirate views, and each candidate must be weighed on their own. To get the pirate thumbs up, you really just have to agree with and support our core values. To give an example of how stuff tends to work, the head of the Oregon Pirate Party is also head of their Green Party (Jorden Leonard.)
So… we started a political party… with no money to speak of… so we started a PAC in Washington… still no money to speak of… oh well, good for us. Let’s put our free time in and see if we can help with anything ongoing, maybe make a difference someday. Even though that sounds less than hopeful, there is actually some light at the end of the tunnel – the one thing the pirate party offers that the other 3rd parties don’t (besides supporting the other 3rd parties) – we win elections with very little money.
It will take a lot of time and resources to really get this off the ground, and the corruption of the system in the US is much more fierce; to make a Starcraft analogy, we see ourselves as the Battle Ships that while you can’t just go rush in from the start with (for the most part) – you can defend your base just long enough to start pumping them out… – Because the corruption must be fought, evil flourishes when good people do nothing.
In the meanwhile, while we were out managing our dicks, the rest of the public started waking up too… realizing that most of what the people in power are saying are lies, that the current media is a corporate owned, for-profit, newertainment industry who’s bias is sold to the highest bidder. And an amazing, truly grass-roots political movement inspired by the Arab Spring, showed up… Occupy Wall Street. First on the internet… and still… even though it has become massively public (seriously look up “Occupy [insert city here]” and you’ll probably find one – it’s on five continents and most US states. There is probably one near you) very empowered by its social presence on the internet… — so you see, protecting the internet, and social media is of profound importance, and even a moral issue at this point.
I was at Occupy Seattle on Friday, September 30th—living in a fairly liberal area and getting off to a late start we haven’t been discriminated against that I know of. I can’t say it’s working like that everywhere. We did not stay the night then, but on Saturday the 1st I did (I didn’t get a very good rest, and had to stop home in the morning to “charge my phone” which later turned into fall asleep until noon. Shame. So I spent most of the day getting my friend geared up for staying overnight – as she has the ability to go live there and be a part of it. I have to work, but I’ll be in and out when I can be and to help. Because we need this.) We held a general assembly of our own both days. There was one on Sunday but I missed it, and the ones today I missed as well (at work), I’m going to see if I can make it tonight anyway to catch up and get the latest news; and tomorrow as well.
Please go and help out if you can, when you can. It changes every day, go to catch up or drop off some blankets or food… — be a part of history.
p.s. I totally got on the news once, my parents called and said they saw me speak. But I can’t find it. I think it was Kiro 7 at 8pm on Oct 1st, if you have a link to it, fwd it to me. I wanna see it!
p.p.s. The WA Pirate Party officially supports Occupy Seattle. And will all do whatever we can to support it and make it work. We will be there. Let us know if there is something we can do for you.
So, if you haven’t been paying attention, several African nations have been involved in revolts lately, and their gov. has been fighting them on every front — including the internet. Keeping information from them, and even using the parts that they leave up and working against them in massive phishing schemes, etc…
In the US we are trying to pass several acts and bills which will bring about new eras in censorship, and tracking, including the ability the gov currently has via warantless wiretapping (enabled by the patriot act) to watch where you are by your cell phone’s radio triangulation (yes, even your dumb phone is betraying you); also including the Cyber Security act they’re trying to pass which would allow them to disable the internet for the private sector (basically the president sends out a message to the ISPs saying quit giving internet to your customers, until further notice.)
A great video on WHY they are doing this: http://vimeo.com/6727469 provided by the pirate party.
Currently to get around censorship there are networks such as Tor, and Freenet and ever growing hordes of encrypted VPN software such as SocialVPN and OpenVPN — these all give security or anonymity of some sort (both good things, as they allow innocent people to spread ideas without persecution.) and some of them use p2p mechanisms.
For the future, I would love to see a truely decentralized internet. One that has no IANA, it uses GUIDs (or Guid pairs) instead of IPv6 addresses; I can already hear you screaming: “but then anyone can change their IP address”, and “but they anyone can spoof anyone else!” — yeah it will need some protections built into it, and it will evolve as it grows (hey just like the current internet!) but if driven by open technologies, with no centralized authority, with the goals of anonymity and security in mind, it would be the “checks and balances” version of the internet. Protected much more from abuse.
But wait you say, “that’s just a logical layer that sits on top of the current internet!”
That’s half true. It sits on top of IP — internet protocol. But, your network doesn’t have to be on the internet to use IP. Among many other types of hardware, you can run IP over 802.11 (common wifi) for short range, up to ~100 yards at 600 Mbps (according to wikipedia, I thought it was 300… the 600 might be an uncommon variant. Need to look into it more.)
100 yards… that’s not much. However, enter stage left: Amatuer Radio; technologies have been in development as long as ARPAnet to develop packet switching over the radio. It’s how modern cell phone technology works. The protocol commonly used for this is apparently 802.16. That technology can be extended to HAM radio.
Upon investigation, it appears that currently it’s not too expensive for a small group of hobbyists to setup a small network of repeaters, where each node has about 6 miles of omni-directional distance (or up to 17 miles line-of-sight/single-directional distance.) — and the bandwidth is around 11 mbits (not great, but better than most home internet connections, and much faster than how the internet 1 started out… see: http://www.qsl.net/kb9mwr/projects/wireless/plan.html )
Lets do this shit
I propose we design, build, test/fine-tune, and deploy back-bone “capsules”.
The capsules would preferably be modular so that they could be configured for different situations accordingly; the general purpose configuration would be a medium powered/refurbed linux laptop like device that bridged a long range radio internet connection with a short range common wifi connection — thus adding stability & distance to the network and also giving out free internet in the local vicinity.
The modularity can also include things like solar panels/batteries (preferred! so that power grid attacks, like what is happening in Egypt today, do not affect the infrastructure), wired connections (to bridge with the actual internet), higher quality antennas, and god knows what else.
We likely will not replace the whole internet with these; but if we wide spread them enough, we can do several things: bring down the cost of internet for everyone, increase our nation’s infrastructure, and most importantly increase our personal ownership of said infrastructure, so that it can be democratically managed, making options such as a kill switch, moot.
As for implementation, we should target secluded areas as well as metro ones, as they usually have the worst internet — Duvall, WA for example has a maximum of 1.5 Mbit ADSL connection to the internet, and likely very little air wave pollution, meanwhile, they have an abundance of IT workers (because they are so close to Redmond, WA), who may be willing to run and maintain these APs.
Secondly, getting local small businesses to run these in metro areas might be easier than you think: They can tell their customers that they give free wireless internet to them, and optionally cancel their own service for one that they don’t have to maintain. The long distance wireless WAN aspect is great because it eliminates much of the upfront ISP costs. People could even build their own hardware and resell the network if they like (think about paying $2.50 a month to your apartment landlord for his maintenance costs and sitting almost directly on an 11mbit “backbone” that is interconnected with the actual internet at various places (likely providing bandwidth much in excess of 11mbits.))
Thirdly, we could put these in our cars, etc — adding to the infrastructure, and giving wifi to our mobile devices in our cars… and those near by. Which also gives the added benefit that some devices like iPods (instead of iPhones) are now (with software like Skype) as good as iPhones because the availability of wifi will be much more wide spread (or even modding your mobile devices’ radio drivers to be able to sit right on that network.)
As a layman I am of the opinion that 802.16 may not be entirely suitable for this, but there is something to be said for open standards and readily available cheap hardware. I think it would be great if people could come up with a cheap, easy to build usb device and provide linux support, possibly using an easy to program microcontroller like an AVR and last-gen antennas that you mind fight used at your local second hand store (such as from a DirectTV set, etc…) so that anyone could join the Broadcasted internet network.
Though, I am concerned that with government projects like HAARP, even wireless APs won’t be good enough to “save us.” if they really want to take down the internet. That’s why we need to fight this battle on multiple fronts: 1.) Fight absurd legislation, even if you lose, you made them work for it, fight and fight hard. We the people need to be heard. and B.) setup as many alternate style networks like these as we can, and interconnect them via any means possible, including the wireless WAN space, the wireless LAN space, and especially the wired internet (especially backbones when possible.)
This will increase the decentralizedness of the internet, add to it’s infrastructure, make it more democratically controlled, and harder even for the government of an area to just go and casually “turn it off” — especially if the majority of such points are controlled by solar. They wouldl need to employ active jamming technologies to completely kill the internet, and/or use physical force.
Thanks for reading!
Check out these two game bundles:
Humble Indy Bundle 2: Pay whatever you want (if you pay more than $7.60 you get Humble Indy Bundle 1 as well!) has several great quality games which are all CROSS PLATFORM and DRM FREE! (Some of them from HIB1 went open source even!)
Braid, Cortex Command, Machinarium, Osmos, Revenge of the Titans, and for Humble Indy Bundle 1 (those who donate over $7.60 currently get these if you don’t already have them): World of Goo, Aquaria, Gish, Lugaru HD, Penumbra Overture, and Samorost 2.
Good deal for under $10, but I recommend giving more.
What’s even cooler is you can divide up how your donation should be spent between the groups and charities which I think is awesome. I plan on donating and helping the EFF because they fight for REAL internet freedom and free speech among other things.
This originally caught my eye on twitter as a post from CymonsGames linking to his blog article about the bundle. The games are by an indy developer who has won some retro remake awards it sounds like. They don’t seem as cool as the Humble Indy Bundle’s in quality, but it’s for a good cause (medical bills?) The games are also DRM free but PC only. Also, it’s a pay what you want kind of deal, so I recommend doing it!
Okay, so I found a pile of games I had written back in the day, and pulled some out and started dusting them off. A couple of guys at work showed some interest and started playing one in particular: Cannon Blaster. I wrote it one day in college when I was bored in Database class ignoring some lecture about sargable queries or some such… Anyway, the game is a quick hack using VB.NET, Winforms, and GDI+ and doesn’t even change your screen resolution… well fast forward to 2010 where we all drive flying cars with bat computers in them and have screens on our cell phones that go up to 20 billion by 15 billion, and well, the game doesn’t play as well… one element in particular: The machine gun doesn’t go far enough to even be usable, you’re stuck with flak only.
Hence Beta 4 (Previous version was labeled as Beta 3). I increased the range of the machine gun (varies by resolution), and *slightly* slowed it down (stops the weapon from giving the player god-like powers). Also there was a bug that when a machine gun hit a plane it didn’t explode, it just dissapeared. I tried to fix that, but the code is pretty much uncommented and super old, so sometimes it works, sometimes it doesn’t, and surprisingly, sometimes it half works…
Anyway, I like the idea behind the game, I might do a more modern version of it that doesn’t blow chunks someday. For now you’ll just have to put up with all the chunk blowing and what not if you want to play it.