Ceph

[Duck Man]

I heard about this through dreamhost a while ago. One of them is actually the guy making it. Anyways, I think it is really cool. So I am sharing it. I am pretty sure serverdm will be seeing this at some point in its life.

[thesi]

And Ceph is what?

[Duck Man]

Lol. I ment to supply a link... http://ceph.newdream.net/

Basically though, its a file system. What makes it different and cool is it allows you to have the filesystem grow over multiple disks, and multiple machines. And it does it all seamlessly.

So lets say we have server dm with an 80 gig drive (thats what it has right now). Now I want more space, so I add a 500 gig drive, I can tie it onto the 80. So now I have 580. I am now garenteed to not be able to lose more than 80 gigs of data. Not very impressive... Now serverdm is maxed out because its using IDE. So I fire up another machine, just for file storage. I but 4 more 500 gig drives. Now I have 2 and half terabytes of data. As long as I keep 500 gigs of space free now I wont lose data if a drive dies. So now I want more space, I can add more drives, or I can add more machines. I can expand infinitely.

Another bonus is that it acts as a raid. So serverdm requests a file, and it reads from its drives, and also receives data from the other computers. My network is setup pretty good right now, but I could add another gigabit card to serverdm, and get another gigabit switch and use it just for the storage. Since most of what I use serverdm for is file serving, this would work perfectly, because I could easily read data faster than I could transfer it out. And that would take up the entire gigabit pipe.

I mean thats pretty bad ass.

Is there anything else we could do with that? Maybe. I don't know how robust it is, but what if along with my little farm that I will eventually have, what if every one of you had a node at your house. Theoretically, you have a small inexpensive mini server with a terabyte of space. Part of my files would be striped onto yours. And some of yours on mine. So you want some of my files, you should get a really good speed compared to just downloading from serverdm directly. And you would have access to 2.5 terabyes of space. Or 5.5 if 3 others also did the same thing.

Now I don't know how well that would actually work. It would have to have some pretty intelligent load balancing. And you would have to have a way to locally mount the filesystem, which I am not sure you can do. But it is a fun senario to think about