ruote tmp/log_2012-11-03.html

2012-11-03 01:03:25 utc mburnett has anyone tried to implement a partial amqp storage component that could be used in a composite storage for messages?
2012-11-03 01:03:53 utc jmettraux not that I know of
2012-11-03 01:04:25 utc jmettraux the way I parse your question, it seems a component that is purely AMQP oriented
2012-11-03 01:04:49 utc jmettraux ... it sounds like a component that ...
2012-11-03 01:05:05 utc mburnett honestly, that's not really true..it's just been a struggle to get ruote-amqp to do my bidding :)
2012-11-03 01:05:19 utc jmettraux sorry, then I don't understand your question
2012-11-03 01:05:21 utc mburnett it just seemed like it might make sense to sort-of leave out the middle man
2012-11-03 01:06:07 utc mburnett well, amqp is not a hard requirement
2012-11-03 01:06:43 utc mburnett being able to submit long running (weeks) jobs to grid and monitor them for failure is
2012-11-03 01:07:25 utc jmettraux what's the grid interface like?
2012-11-03 01:07:30 utc jmettraux create(job)
2012-11-03 01:07:32 utc mburnett DRMAA
2012-11-03 01:07:33 utc jmettraux status(job_id)
2012-11-03 01:07:37 utc mburnett basically
2012-11-03 01:07:39 utc jmettraux cancel(job)
2012-11-03 01:08:03 utc jmettraux so you have to poll the grid for finished jobs
2012-11-03 01:08:05 utc jmettraux ?
2012-11-03 01:08:13 utc jmettraux no callbacks?
2012-11-03 01:08:29 utc mburnett well, we can either poll for job status or wrap the sub'd commands in something that phones home occasionally and when finished
2012-11-03 01:08:45 utc mburnett the callbacks are bad..they rely on your process staying alive
2012-11-03 01:08:50 utc mburnett which for weeks is silly
2012-11-03 01:09:03 utc mburnett by process, i mean the monitoring process
2012-11-03 01:09:09 utc jmettraux so you have to write a middleman anyway
2012-11-03 01:09:22 utc jmettraux keeping a list of open jobs and asking for them occasionally
2012-11-03 01:09:28 utc mburnett yes
2012-11-03 01:09:42 utc mburnett hence interest in amqp
2012-11-03 01:09:53 utc mburnett also, some others at the center are interested in it for other purposes
2012-11-03 01:10:22 utc mburnett so, i thought to myself today that it might be easier to just have ruote pass all it's messages through amqp, but maybe that's not true
2012-11-03 01:10:24 utc jmettraux I guess one can put back a message with a delay
2012-11-03 01:10:54 utc jmettraux you mean have ruote communicate with its participants with amqp in between?
2012-11-03 01:11:35 utc mburnett well, i guess i just mean have get/put_msg in the storage use amqp and the rest of storage use redis
2012-11-03 01:11:46 utc mburnett that way i don't need an "adapter" participant
2012-11-03 01:11:56 utc mburnett that catches a ruote message, then just writes an amqp message
2012-11-03 01:13:01 utc mburnett also, i'm a little concerned about handling certain race conditions, where e.g. i get a message to submit a job, but then i crash before i can actually submit the job
2012-11-03 01:13:10 utc jmettraux you mean use amqp as the "ruote msg" queue?
2012-11-03 01:13:18 utc mburnett yes
2012-11-03 01:13:34 utc jmettraux why not
2012-11-03 01:13:47 utc mburnett so you think it's a reasonable idea?
2012-11-03 01:13:50 utc jmettraux no
2012-11-03 01:13:53 utc mburnett oh lol
2012-11-03 01:14:26 utc mburnett i must just misunderstand then :)
2012-11-03 01:14:58 utc jmettraux it's possible
2012-11-03 01:15:12 utc jmettraux but those messages are "ruote only"
2012-11-03 01:15:28 utc mburnett sure
2012-11-03 01:15:33 utc jmettraux of course some observers might benefit from peeking at them
2012-11-03 01:16:00 utc jmettraux but well
2012-11-03 01:16:05 utc mburnett and i suppose it's not possible to have a worker with only some of the possible participants consuming those messages
2012-11-03 01:16:19 utc jmettraux it's possible
2012-11-03 01:16:35 utc jmettraux but not optimal
2012-11-03 01:16:51 utc jmettraux (not sure if we're talking about the same things)
2012-11-03 01:17:02 utc mburnett it feels like we might be
2012-11-03 01:17:51 utc mburnett for example, we will (at least for a while) have some participant activity that requires a large set of perl modules
2012-11-03 01:18:13 utc mburnett so the current plan is to just have amqp participants that send orders to listeners with those modules already loaded
2012-11-03 01:18:20 utc jmettraux sounds right
2012-11-03 01:19:52 utc jmettraux so some of the work is done in the grid, some not
2012-11-03 01:20:51 utc jmettraux ok, have to go, good evening!
2012-11-03 01:20:51 utc mburnett yep
2012-11-03 01:20:57 utc mburnett thanks for all your help!
2012-11-03 01:21:07 utc jmettraux you're welcome!