| 2012-11-03 01:03:25 utc | mburnett | has anyone tried to implement a partial amqp storage component that could be used in a composite storage for messages? |
| 2012-11-03 01:03:53 utc | jmettraux | not that I know of |
| 2012-11-03 01:04:25 utc | jmettraux | the way I parse your question, it seems a component that is purely AMQP oriented |
| 2012-11-03 01:04:49 utc | jmettraux | ... it sounds like a component that ... |
| 2012-11-03 01:05:05 utc | mburnett | honestly, that's not really true..it's just been a struggle to get ruote-amqp to do my bidding :) |
| 2012-11-03 01:05:19 utc | jmettraux | sorry, then I don't understand your question |
| 2012-11-03 01:05:21 utc | mburnett | it just seemed like it might make sense to sort-of leave out the middle man |
| 2012-11-03 01:06:07 utc | mburnett | well, amqp is not a hard requirement |
| 2012-11-03 01:06:43 utc | mburnett | being able to submit long running (weeks) jobs to grid and monitor them for failure is |
| 2012-11-03 01:07:25 utc | jmettraux | what's the grid interface like? |
| 2012-11-03 01:07:30 utc | jmettraux | create(job) |
| 2012-11-03 01:07:32 utc | mburnett | DRMAA |
| 2012-11-03 01:07:33 utc | jmettraux | status(job_id) |
| 2012-11-03 01:07:37 utc | mburnett | basically |
| 2012-11-03 01:07:39 utc | jmettraux | cancel(job) |
| 2012-11-03 01:08:03 utc | jmettraux | so you have to poll the grid for finished jobs |
| 2012-11-03 01:08:05 utc | jmettraux | ? |
| 2012-11-03 01:08:13 utc | jmettraux | no callbacks? |
| 2012-11-03 01:08:29 utc | mburnett | well, we can either poll for job status or wrap the sub'd commands in something that phones home occasionally and when finished |
| 2012-11-03 01:08:45 utc | mburnett | the callbacks are bad..they rely on your process staying alive |
| 2012-11-03 01:08:50 utc | mburnett | which for weeks is silly |
| 2012-11-03 01:09:03 utc | mburnett | by process, i mean the monitoring process |
| 2012-11-03 01:09:09 utc | jmettraux | so you have to write a middleman anyway |
| 2012-11-03 01:09:22 utc | jmettraux | keeping a list of open jobs and asking for them occasionally |
| 2012-11-03 01:09:28 utc | mburnett | yes |
| 2012-11-03 01:09:42 utc | mburnett | hence interest in amqp |
| 2012-11-03 01:09:53 utc | mburnett | also, some others at the center are interested in it for other purposes |
| 2012-11-03 01:10:22 utc | mburnett | so, i thought to myself today that it might be easier to just have ruote pass all it's messages through amqp, but maybe that's not true |
| 2012-11-03 01:10:24 utc | jmettraux | I guess one can put back a message with a delay |
| 2012-11-03 01:10:54 utc | jmettraux | you mean have ruote communicate with its participants with amqp in between? |
| 2012-11-03 01:11:35 utc | mburnett | well, i guess i just mean have get/put_msg in the storage use amqp and the rest of storage use redis |
| 2012-11-03 01:11:46 utc | mburnett | that way i don't need an "adapter" participant |
| 2012-11-03 01:11:56 utc | mburnett | that catches a ruote message, then just writes an amqp message |
| 2012-11-03 01:13:01 utc | mburnett | also, i'm a little concerned about handling certain race conditions, where e.g. i get a message to submit a job, but then i crash before i can actually submit the job |
| 2012-11-03 01:13:10 utc | jmettraux | you mean use amqp as the "ruote msg" queue? |
| 2012-11-03 01:13:18 utc | mburnett | yes |
| 2012-11-03 01:13:34 utc | jmettraux | why not |
| 2012-11-03 01:13:47 utc | mburnett | so you think it's a reasonable idea? |
| 2012-11-03 01:13:50 utc | jmettraux | no |
| 2012-11-03 01:13:53 utc | mburnett | oh lol |
| 2012-11-03 01:14:26 utc | mburnett | i must just misunderstand then :) |
| 2012-11-03 01:14:58 utc | jmettraux | it's possible |
| 2012-11-03 01:15:12 utc | jmettraux | but those messages are "ruote only" |
| 2012-11-03 01:15:28 utc | mburnett | sure |
| 2012-11-03 01:15:33 utc | jmettraux | of course some observers might benefit from peeking at them |
| 2012-11-03 01:16:00 utc | jmettraux | but well |
| 2012-11-03 01:16:05 utc | mburnett | and i suppose it's not possible to have a worker with only some of the possible participants consuming those messages |
| 2012-11-03 01:16:19 utc | jmettraux | it's possible |
| 2012-11-03 01:16:35 utc | jmettraux | but not optimal |
| 2012-11-03 01:16:51 utc | jmettraux | (not sure if we're talking about the same things) |
| 2012-11-03 01:17:02 utc | mburnett | it feels like we might be |
| 2012-11-03 01:17:51 utc | mburnett | for example, we will (at least for a while) have some participant activity that requires a large set of perl modules |
| 2012-11-03 01:18:13 utc | mburnett | so the current plan is to just have amqp participants that send orders to listeners with those modules already loaded |
| 2012-11-03 01:18:20 utc | jmettraux | sounds right |
| 2012-11-03 01:19:52 utc | jmettraux | so some of the work is done in the grid, some not |
| 2012-11-03 01:20:51 utc | jmettraux | ok, have to go, good evening! |
| 2012-11-03 01:20:51 utc | mburnett | yep |
| 2012-11-03 01:20:57 utc | mburnett | thanks for all your help! |
| 2012-11-03 01:21:07 utc | jmettraux | you're welcome! |