#gridfs

AngularアプリケーションでCKEditor5のSimpleUploadAdapterを使い、画像をサーバーにアップロードするものを作ってみました。説明を書くと長くなるので、解説文をGistに載せました。

こんな感じで動きます。

CKEditorの画像ボタンをクリックして画像ファイルを選択。

アップロードが完了したときの画面。

MongoDB provides a special specification named GridFS for storing and retrieving files such as images, audio files, video files, etc that exceed the BSON-document size limit of 16MB. It is kind of a file system to store files but its data is stored within MongoDB collections.

GridFS basically divides each file into several parts or chunks, instead of storing the file in a single document and each chunk is stored as a separate document. And the limits for the chunk size is 255k. GridFS uses two collection to store files: • Chunks collection(which stores the binary chunks) • Files collections(store the file’s metadata)

Chunks Collection: In the chunk collection, each document represent the distinct chunk of a file as represented in the GridFS. Following is the example to represent the chunk collection: { “_id” :<objectid>, “Files_id” :<objectid>, “number” :<num>, “data” :<binary> } Files Collection: Each document present in the files Collection represents a file in the GridFS. Following is the example of file collection. {  "_id" : <ObjectId>,  "length" : <num>,  "chunkSize" : <num>,  "upload_Date" : <timestamp>,  "md5" : <hash>,  "file_name" : <string>,  "content_Type" : <string>,  "aliases" : <string array>,  "metadata" : <dataObject>, }

How to add files to the GridFS? Now to store the mp3 file using GridFS using the Put command. For this use the mongofiles.exe utility present in the bin folder of the MongoDB installation folder. Open the command prompt, navigate to the mongofiles.exe in the bin folder of the MongoDB installation folder and type the command: >mongofiles.exe -d GridFS put Song.mp3 Adding an mp3 file to GridFS:

In the chunk collection, each document represent the distinct chunk of a file as represented in the GridFS. Following is the example to represent the chunk collection

To show the mp3 file connect to the mongo server and navigate through the databases, use the gridfs database and execute the find() command inside the fs.files collection. >db.fs.files.find()

Advantages of GridFS:

• Helps to overcome file system limitation like storing large number of files. • For accessing selected portion of the files. • Storing user generated file content • Storing those documents whose size is greater than the 16MB. • Storing the files along with the databases and eradicating the problems like:  Replicating the files to all needed servers  How to delete the copies when the file is deleted? etc.

Today I discovered how easy it is to send a file over the web sockets directly to MongoDB GridFS. It is possible thanks to two amazing node modules. One of them is Naoyuki Kanezawa's Socket.IO stream. The other one is Aaron Heckmann's gridfs-stream.

I will get right to the point. I will just assume you read the documentation of both modules, and included them into your Node.js project. The client side code will be no different then the example in Socket.IO stream project documentation. This is example how receiving of the file and writing it to GridFS could look like:

io.sockets.on('connection', function (socket) { ss(socket).on('uploadFile', function (stream, data, callback) { var writestream = gfs.createWriteStream(); stream.pipe(writestream); }); ... });

You need the following gems (in addition to your mongomapper, sinatra e.t.c) to accomplish this:

joint

rack-gridfs

Add an attachment to your model like so:

class MyDocument include MongoMapper::Document plugin Joint key :name, String, :required => true attachment :logo end

With that done, you can now save an image to the database like so:

logo = File.open(Dir.pwd + "/your_file.png") doc = MyDocument.create(:name => 'Test',:logo => logo) image_id = doc.logo.id

In order to retrieve the images you need to attach the gridfs end point so that it knows to serve images from gridfs

class MediaApp < Sinatra::Base db = 'my_db' use Rack::GridFS, :database => db, :prefix => 'images' get /.*/ do "The URL did not match a file in GridFS." end end

(See an earlier post on how to get the name of your mongo db database from a heroku MongoHQ url, if you're using heroku).

Now we need to mount the rack up by adding this app into the config.ru

map "/media" do run MediaApp end

With that you can now retrieve the image from the id of the item (image_id from the second listing). The url to access the 

http://<your_server>:<port>/media/images/<image_id>

Aristarkh Zagordnikov wrote me an email describing the reasons that led his company create and open source mod_gridfs.

Some time ago we were looking for a way to serve files to the web right from the GridFS database. We considered different options, including IIS handler (we use .NET on Windows as a backend) that requires a Windows machine to serve files (we planned to use Windows as backend only), nginx-gridfs that was too slow (because it’s synchronous and nginx isn’t, and uses the not-very-much-up-to-date MongoDB C driver that doesn’t do connection pooling, etc.) and does not support slaveOk (horizontal sharding).

At last I decided to roll our own method: a module for Apache 2.2 or higher that uses MongoDB’s own C++ driver. It supports replica sets, slaveOk reads, proper output caching headers (Last-Modified, Etag, Cache-Control, Expires), properly responds to conditional requests (If-Modified-Since/If-None-Match), and uses Apache brigade API to serve large files with less in-memory copying.

While Apache isn’t the most resource-friendly server for a high-load environment (it consumes too much memory per connection and does not yet support production-quality event-based I/O), it really shines as a backend for something like nginx+proxy_cache with optional SSD as proxy_cache storage that does the heavy lifting.

Serving a 4KiB file over a gigabit network on modern hardware, 100 concurrent requests, MongoDB replica set of 3 machines as a backend:

NGINX + nginx-gridfs: 1.2kr/s

Apache + mod_gridfs: 6.6kr/s

Apache + mod_gridfs with slaveOk: 12.1kr/s

I didn’t test with larger files, because this way I’ll be benchmarkng OS I/O performance instead of user-mode code.

The public Mercurial repo is here. It uses Simplified 2-clause BSD license, and contains installation instructions and docs in the README file (building might seem hard, but after building if you have to mass-deploy, you just install dependent libraries like boost and copy the mod_gridfs.so file around).

Contrary to many MongoDB deployments, we primarily use it for storing files in GridFS. We switched over to MongoDB after searching for a good distributed file system for years. Prior to MongoDB we used a regular NFS share, sitting on top of a HAST-device. That worked great, but it didn’t allow us to scale horizontally the way a distributed file system allows.

No doubt GridFS is a useful feature of MongoDB, but I’m pretty sure the experts in distributed file systems have better solutions for this—I just hope they’ll share it with us.

Update: Jeff Darcy1:

Yes, we do have better solutions for this particular kind of use case.  So do object/blob stores like Swift.  

Honestly, I don’t think the “searching for a good distributed filesystem” part is even credible. How can someone be that bad at finding readily available information?  For example, it’s easier to set up sharding and replication with GlusterFS than with MongoDB and GridFS, plus you’ll get striping and RDMA and generally better performance for this type of workload.  On top of all that, you won’t need to use special libraries to interface with it because it’s a regular POSIX filesystem.  Lastly, it’s not like there hasn’t been a lot of press about it.  Even considering their obvious FreeBSD bias and the fact that FreeBSD is weak in this area, the second i tem for “FreeBSD distributed filesystem” points to GlusterFS.  If they didn’t find it, they just didn’t look very hard before they reached for the New Shiny.  

It’s not just GlusterFS, either.  MogileFS might not be a real filesystem but it’s user space so it would probably run just fine in their environment - as would the aforementioned Swift.  I have more of a problem with the anti-Mongo haters than with Mongo itself, it’s wonderful that these guys found a Mongo-based solution that works for them, but it seems like a bit of an odd choice nonetheless.

Jeff Darcy is a member of the gluster.org advisory board, and works on GlusterFS full time at Red Hat. He’s also the person I direct all my questions related to distributed file systems (and not only). ↩