ducktyped - Home

The great Ruby web framework multi-app challenge

2008-01-04T00:11:00Z

Many websites are composed of several fairly independent apps that would still benefit from tighter integration than running them as separate apps. Here's a typical site: core site, forum, blog/cms (or photo management app, etc).

The typical Rails response to multi-app integration is one of the following two: 1. "Just run them as separate apps on different subdomains and share the database." or 2. "Use the app as the base and add your own code to it." I have integrated several Rails apps, and it is dirty and inefficient, a real pain in the ass, especially compared to the joy of doing most other things with Ruby/Rails. The failure of multi-app Rails is documented by the plethora of hacks used to emulate multi-app behaviors: generators, engines, appable plugins, and the multi-app routing plugin.

The point of this post is not to rip on rails, it's just what I'm primarily using, and I would love to see other Ruby web frameworks such as merb, ramaze, and the micro-frameworks think about multi-app while they can still make major changes. Ruby is powerful enough to elegantly enable multi-app configurations.

What sucks about multi-app integration right now

Redundant css and js

You send a new copy of prototype.js when switching between your core app, beast, and mephisto for example. You duplicate CSS as well. To avoid that you end up futzing around with deployment managers (capistrano) to make one app use the other's css and js in production.

Duplicate models

You have to duplicate models to access resources across the separate apps, or set up SCM, symlink hacks for versioning. Ugly, and suckage for dev / deployment differences.

Shitty routing

You end up using hacks and the web server instead of the app's router to resolve your routing. It's less powerful and in the wrong place. Now there is the multi-app routing plugin, which is just a hack to mitigate an underlying bigger problem. (It's cool for some use cases, sure, but still limited in capabilities)

Code scales poorly

Dumping all apps' M/V/Cs into one big app creates a cluttered project, and the project code doesn't scale. Here's a post by caboose's Courtenay hinting at that the rails directory structure struggles with large apps. Having "eighty controllers" should not be a reason for having to split a project into two completely separate projects, maintaining separate deployments, etc.

In a way, the Rails folder structure is backwards, e.g. I can namespace my M/V/Cs but that means I have pieces of a semantically related structure (an app) in 3 places instead of one. When I work on code, I mostly work on a subcomponent of the app, and not just in M or V or C.

What I want is:

/app1
  /model
  /view
  /controller
/app2
  /model
  /view
  /controller

Not:

/model
  /app1
  /app2
/view
  /app1
  /app2
/controller
  /app1
  /app2

Memory overhead

People on shared hosts and small VPSes don't have ram to throw around. Right now, with a rails site consisting of a core app, blog, and forum, I get 3 hefty rails processes, when I want just one. (I could then use two mongrels, save 1/3 ram and get better concurrency/basic load balancing (if one app gets hit much more than the other)).

Single sign-on and session/cookie sharing

This isn't too hard to solve in practice, but it's not necessary. There should be a auth/login app to drop in instead of the generator. Other apps then by convention use user and user_id as the hooks into the auth/login system, but they don't all duplicate it. The auth/login system can be whatever, as long as the interface to it is a user model and user_id as the hook. All that would be left to do to integrate it is to add associations and before_filters. This has been a big topic of debate in the beast forums, and frankly, it shouldn't be that integrating a rails app with beast (another rails app) is exactly as cumbersome as integrating it with a PHP forum engine.

Fighting objections - it's all about views

Plugins already provide (80% of) this functionality

No they don't, and they shouldn't. Engines, appable plugins, and generators are all trying to fix part but not the root of the multi-app problem.

Most importantly, plugins have no views. Have your ever counted your LOC? Views make up a lot of code, especially with ajax thrown in. For example, I want to be able to drop in a blog or comment system that includes ajax posting of comments by default, rather than me having to write all the views from scratch. Plugins are fantastic for what they're designed for, but multi-app is not their use case.

High level components don't work

DHH is a declared opponent of components (or at least was, given that the quoted statements are over 2 years old). (see here and here).

His experience is based on a failing high level parameterized component architecture in java. But the components he's talking about are different from multi-app. I'm not saying that you can just drop in a bunch of components and wire them together with a few parameters. I'm saying, get canonical code (e.g. a basic blog engine) and adapt it to your needs in a way that's cleaner and more maintainable/reusable than a generator's output or nested plugin stuff like engines or appable plugins.

Also, David's experience was based on working in a java shop, but Ruby is much more concise than Java, and modifying a forum engine in Ruby isn't too bad, especially if its code layout follows a canonical layout such as the standard rails app structure.

Ruby/Rails makes it so easy to write from scratch, why bother?

Why would you need this stuff, if Ruby/Rails make it so easy to create a blog engine from scratch. DHH did it in 10 minutes in his video, Akita in 20 minutes. Did they really?

It's like saying "I can write a digg clone in 20 minutes." You can write something that provides the most basic functionality, but it's not a digg clone. You know why? UI, UI, UI. User interface is 80% of the value of an app for the end user, and right now there is no reuse of great UI design, which is a shame, because UI design is hard, and a lot of people would benefit from leveraging great UIs. I'd rather drop a mephisto-lite into my project than rewrite it from scratch to "meet my app specific needs". And I want to do it without mixing its code and my own apps code wildly all over the place (which just creates SCM headaches).

What could multi-app look like?

The one framework I found that gets multi-app mostly right is Python's django. It's great to be able to drop in a basic blog or forum engine that includes views/ajax, etc.

At the very least, it makes utter sense to have common routing, css, javascript for a collection of apps in a site.

DRY up the suckage described above

CSS, JS, models
Sessions/cookies
Routing
(Single sign-on)

Rough idea for a project/app layout

/app_root
    /config (global configuration, affects all apps)
    /public
        /js (main app default)
        /css 
        /blog/js (blog specific)
        /blog/css
        /forum/js
        /forum/css
    # These are the apps that would be dropped into the application
    # Test and spec folders would be moved into the app folders, so
    # that each app can have its own test suite
    /app (The default main app)
    /blog (Full app with tests, etc., minus config, public)
    /forum (same thing)

How to adjust routing

Routing could simply add a an 'app' level :app/:controller/:action/:id => http://mysite.com/forum/post/view/1

Problems to address

Layouts and css

How to avoid layout duplication, and where to put app-specific css. I would put css in public/css/#{app_name}/default.css, and then for deployment join the css files. This would also require some name-spacing conventions of css to avoid collisions/overrides. I haven't fully thought through how to do a DRY layout structure.

DB table name collisions

I like django's solution of prefixing tables with the app name, e.g. a posts model in a blog and a forum app would map to the following tables:

blog_posts
forum_posts

This could be easily done by using the ORMs' table prefix methods, and could be automated, dealt with by convention.

Different apps being tied to different versions of the framework

What if my core app runs on edge, but my dropped-in mephisto-lite relies on Rails 1.2.x. Since I'm now running within the same process, this wouldn't be possible AFAIK, but the tradeoff is still worth it. Especially if you reuse a lot of your own internal apps. It might even lead to better maintained apps.

I made the assumption of everything running in one process before, but that wouldn't necessarily have to be the case. Apps could run in their own processes with their own version of the framework, but I still have a smart wrapper around them for routing, etc.

Apps use different ORMs or databases

I haven't thought through this one too much, but it's even conceivable that apps could use different ORMs to talk to the database, or even use separate databases. I would probably have a global default db config, with possibilities for apps to override them. That might make the framework startup process overly complex, so I'd even be fine if all apps would have to use the same ORM/db. (Side note: Multi-db is still another unsolved challenge for most Ruby web frameworks).

Some folder locations might be contentious

Should /lib be global or per-app? Per-app keeps things more in one place and allows different apps to use different versions of the same library if necessary. Perhaps a structure of /lib/app_name would solve this in an elegant way.

Conclusion

There is movement in the Ruby web framework space with new frameworks hitting the scene. "The balls are in the air", and now is the time to bring up ideas for change. Rails has been and still is a fantastic yardstick by which to measure web frameworks, but Rails has warts too, and sometimes Rails is starting to feel a bit pudgy, like a "Ruby Struts" (which is amazing, because Rails is at least an order of magnitude better than Java Struts, it just shows how much progress is being made.)

I talked about multi-app in this post based on my personal experience with Rails. Rails is fantastic, but I don't think they would incorporate a clean multi-app setup, but I'm hopeful that the new kids on the Ruby web framework block (like merb, ramaze, sinatra, et al.) will think hard about if and how to incorporate multi-app capabilities. Especially some of the micro-frameworks could scale very nicely (code management and deployment-wise) by clustering small, light apps together (like a bliki for example).

I hope to stimulate a healthy, productive, and civilized discussion with this post. Hopefully, at least one framework will incorporate multi-app capabilities. Whichever Ruby web framework will end up with a clean multi-app setup will be the one I'll be using.

Merb startup problems: dies after "Compiling routes.."

2007-12-10T11:55:00Z

I was so excited to play around with merb again. I just checked, and the first version of merb I used to prototype an app was 0.0.8 about a year ago. But I was so bummed out that after wanting to be all proper and upgrading all my gems, merb stopped working, no errors, no logs, no debug info, nothing.

After asking on IRC (impossibly late at night) without any luck, I dove into the (beautifully clear and manageable) source and figured out that the ‘parsetree’ gem was required, but not installed. So a quick sudo gem install parsetree, was all that I needed, and I’m back in business.

After I realized that parsetree was to blame, a quick search brought up the same problem addressed and solved on the Ruby mailing list: http://www.ruby-forum.com/topic/132766. But without knowing that the problem is parsetree related, how would you find this post. So again, for the crawlers merb 0.4.1 stops after “Compiling routes..”

I can’t wait to get some real work done now. :)

Set up IE testing on localhost using Parallels on OS X

2007-07-02T10:57:00Z

After an extensive hunt for a low-hassle solution for testing my local development rails applications in Internet Explorer using Parallels, I finally found this post.

This is by far the most straightforward way and clearest writeup to get this development plumbing out of the way. It uses Bonjour for Windows so that you can access your rails apps with something like mymachine.local:3000. Done! Now back to actual coding.

Get SVN changesets by RSS for any public repository

2007-06-16T10:54:00Z

Subtlety is a beautiful thing. Just enter the repository’s url and add the resulting subtlety link to your feed reader. I’ve never enjoyed reading changesets so much. I actually follow projects now without missing anything.

How to change databases using Ruby on Rails

2007-06-12T08:09:00Z

I was running a mysql 4.1 db that kept screwing up my unicode, so I decided why not double version numbers and move to postgresql 8.2. Great idea in theory, but a pain in the ass in practice. The sql dumps of the different DBMS are incompatible, especially booleans causing trouble. The scripts I found were pretty much useless, so I was stuck until I came across the idea of using rails to export the db to yaml and then re-import it into the new db.

This sounds very elegant in theory, but there are still a few pitfalls. Tobias Luetke posted how he did it and provided a very useful rake task that became the basis for my approach. For good measure I threw Dr. Nic’s magic model generator in the mix to allow for very powerful and flexible migrations, even if your application doesn’t define models for all tables, e.g. when you have several apps using the same db.

The following is my 10-step program for database migration bliss.

Set up the basics


  sudo gem install magic_model_generator
  rails db_migration_app
  cd db_migration_app

Download the backup.rake task and copy it to lib/tasks/
Connect to your production db in config/database.yml. This is also a good time back up your db before you ruin things with jittery fingers and then blame me.
With the db set up properly, get the schema and magic models ready
```
  RAILS_ENV=production rake db:schema:dump
  RAILS_ENV=production ./script/generate magic_model
  
```
(Just for kicks, you can inspect your entire db through the rails console via RAILS_ENV=production ./script/console)
[OPTIONAL] Add any tables that should not be dumped/restored to the array in the rake task’s interesting_tables method. schema_info, sessions, and logged_exceptions are excluded by default.
Let’s do some dumping. This may take some time, depending on your db size.
```
  RAILS_ENV=production rake db:backup:write
  
```
At this point you have all your tables’ data in yaml format in db/backup/*.yaml.
Now change your database.yml to the new, empty db (e.g. your brand new pg db). Don’t mess this up, because the next step will completely wipe whichever db is defined as the production db.
Import the yaml data into the new production db.
```
  RAILS_ENV=production rake db:backup:read
  
```
This also runs rake db:schema:load, and restores the correct schema_info.
[EXTRA CREDIT] If you migrate to postgresql, fix your sequences in one easy step with the following rake task I created:
```
  RAILS_ENV=production rake db:backup:set_sequences
  
```
If you want to check the sequence number in psql, run select last_value from table_name_id_seq; (where table_name is your table’s name)
Lastly, make sure you don’t forget to adjust your actual application’s database.yml to point to the new db.

Enjoy your newfound freedom from the stranglehold of your old database. :)