| 2012-11-20 14:52:52 utc | mburnett | jmettraux: I know you read the logs, so I'll just respond now :) |
| 2012-11-20 14:53:35 utc | mburnett | i'm happy to send you my slides from yesterday so you can see what kind of architecture we're looking at..it's essentially a rabbitmq centered approach |
| 2012-11-20 14:54:22 utc | mburnett | i suspect what we will do is implement all the services we care about and use rabbit as an intermediary to ruote and then replace ruote if it is still not performant enough |
| 2012-11-20 14:55:04 utc | mburnett | we already have 3 very basic competing implementations..the most promising doesn't use sidekiq or resque at all (they are unreliable and un scalable compared to amqp) |
| 2012-11-20 14:55:34 utc | mburnett | i do think that it would be easy to under estimate the amount of effort for a complete feature set |
| 2012-11-20 14:55:58 utc | mburnett | the first thing we're going to look at is improving ruote to be able to meet our needs (if it can't already) |
| 2012-11-20 14:56:57 utc | mburnett | the rabbitmq + redis prototype that we implemented can handle 10k concurrent jobs in something like 22 seconds |
| 2012-11-20 14:57:14 utc | mburnett | that's with 1 worker, and it appears to scale very well |
| 2012-11-20 14:58:32 utc | mburnett | it also seems to have excellent crash recovery essentially out of the box (amqp makes reliable messaging much easier than redis/resque) |
| 2012-11-20 20:54:26 utc | jmettraux | mburnett: thanks! |
| 2012-11-20 20:57:38 utc | lmarburger | mburnett: wow! 10k concurrent jobs? that's awesome! |
| 2012-11-20 20:58:37 utc | lmarburger | i haven't had the need for any "real" queueing. things are still small enough that a database-backed queue is a great option. |
| 2012-11-20 21:02:10 utc | jmettraux | lmarburger: hello, I think that the rabbitmq + redis prototype mentioned by mburnett doesn't include ruote ;-) |
| 2012-11-20 21:03:00 utc | lmarburger | oh ha! i thought that was rabbitmq + redis that was being fed jobs by ruote. |
| 2012-11-20 21:03:27 utc | jmettraux | ah, that could make sense |
| 2012-11-20 22:00:42 utc | mburnett | jmettraux: is there a dtd for the xml format? |
| 2012-11-20 22:00:50 utc | mburnett | of process definitions |
| 2012-11-20 22:01:00 utc | jmettraux | mburnett: sorry, none at all |
| 2012-11-20 22:01:04 utc | mburnett | ok |