ruote tmp/log_2011-07-01.html

2011-07-01 07:39:56 utc jmettraux tosch_le: hello
2011-07-01 07:40:16 utc tosch_le hello!
2011-07-01 07:41:10 utc tosch_le started working on a fsstorage-only worker yesterday, but i'm still unsure about the way to go.
2011-07-01 07:41:25 utc jmettraux the other day, I wanted your opinion on relaxing the dependencies for ruote-kit
2011-07-01 07:41:44 utc jmettraux fs_storage: sweet
2011-07-01 07:42:24 utc tosch_le had a look at https://github.com/ttilley/fssm
2011-07-01 07:42:48 utc jmettraux looks ideal
2011-07-01 07:43:34 utc tosch_le my only worry is that i'll need two threads: one for fssm and one for checking the schedules.
2011-07-01 07:43:55 utc tosch_le there might be issues when there's no file locking mechanism
2011-07-01 07:44:14 utc tosch_le or not?
2011-07-01 07:44:15 utc jmettraux what about having 1 thread that receives msgs and schedules events ?
2011-07-01 07:44:34 utc jmettraux (it would have to keep all the schedules in memory though)
2011-07-01 07:45:34 utc tosch_le i can't change the loop fssm provides as it's dependent on the backend fssm uses, so that'll be hard
2011-07-01 07:45:47 utc tosch_le i thought about using https://github.com/mockko/em-dir-watcher
2011-07-01 07:45:59 utc tosch_le and using em for the schedules, too
2011-07-01 07:46:15 utc jmettraux you can watch a directory ?
2011-07-01 07:46:33 utc tosch_le using em-dir-watcher, yes.
2011-07-01 07:46:56 utc jmettraux FSSS
2011-07-01 07:47:44 utc jmettraux glob **/msg-*
2011-07-01 07:47:51 utc jmettraux ah yes
2011-07-01 07:47:53 utc jmettraux :-(
2011-07-01 07:48:26 utc tosch_le pardon, what does fsss mean?
2011-07-01 07:48:40 utc jmettraux sorry, was looking at fssm doc
2011-07-01 07:48:57 utc jmettraux we could move msgs and schedules in a dir
2011-07-01 07:49:03 utc jmettraux so that we could watch that dir
2011-07-01 07:49:11 utc jmettraux and not care about other changes
2011-07-01 07:49:32 utc tosch_le they are in separate dirs and that's lovely and sufficient.
2011-07-01 07:50:15 utc jmettraux but one watch thread is better than two ?
2011-07-01 07:50:40 utc tosch_le one watch thread should be able to watch more than one dir
2011-07-01 07:51:12 utc jmettraux so life is good
2011-07-01 07:51:33 utc tosch_le but that doesn't solve the schedules problem: when it's time for running a schedule, no change in the fs is made and so no event is fired
2011-07-01 07:51:58 utc jmettraux that's why you'd be forced to keep all the schedules in memory
2011-07-01 07:52:02 utc tosch_le yes
2011-07-01 07:52:09 utc jmettraux and update that table when new schedules come in
2011-07-01 07:52:12 utc jmettraux (or go out)
2011-07-01 07:52:39 utc jmettraux when the time comes, you reserve the schedule, trigger it and life is good
2011-07-01 07:53:15 utc tosch_le yeah, but one thread will have to make sure there is a trigger when the time comes, having the schedules in memory or not.
2011-07-01 07:53:32 utc jmettraux ah yes
2011-07-01 07:53:37 utc jmettraux two threads
2011-07-01 07:54:32 utc jmettraux schedules are tricky
2011-07-01 07:54:52 utc tosch_le and i think i was wrong about the file locking problem: schedules are not msgs, when one thread triggers a schedule, there should be no conflict with any msg being processed by the other thread
2011-07-01 07:55:10 utc jmettraux ok
2011-07-01 07:56:06 utc tosch_le schedules are tricky, yes, that's why using eventmachine was appealing to me: i'd have events for fs changes and events for schedules, too.
2011-07-01 07:56:45 utc jmettraux sounds good
2011-07-01 07:57:36 utc tosch_le and fssm has the drawback that it lacks a proper #stop method. it just waits for an interrupt and that's somewhat annoying.
2011-07-01 07:58:09 utc jmettraux can't remove the trap ?
2011-07-01 07:59:01 utc tosch_le the run loop is something like "loop while 1 rescue Interrupt"
2011-07-01 07:59:24 utc tosch_le no way to stop it besides raising an interrupt
2011-07-01 08:03:09 utc tosch_le anyway, i think i'll try the eventmachine way: i'll watch the msgs dir and process msgs when they arrive. i'll use EM::Timer for the schedules and create those timers through watching the scheds dir for changes.
2011-07-01 08:03:15 utc jmettraux the em watcher works on linux + osx ?
2011-07-01 08:03:16 utc tosch_le or some way like that.
2011-07-01 08:03:42 utc tosch_le windows is supported, too.
2011-07-01 08:03:58 utc tosch_le ( via win32-changenotify)
2011-07-01 08:04:00 utc jmettraux schedules : once per minute ?
2011-07-01 08:04:21 utc tosch_le no, instantly when it's time for them.
2011-07-01 08:04:38 utc tosch_le they'll be in memory.
2011-07-01 08:04:41 utc jmettraux so it's limited to 1 worker
2011-07-01 08:04:50 utc jmettraux ah ok
2011-07-01 08:05:29 utc jmettraux sounds nice
2011-07-01 08:05:46 utc tosch_le let's see where i get with this.
2011-07-01 08:06:15 utc tosch_le anyway, back to ruote-kit. which dependencies do you want to remove/change?
2011-07-01 08:06:19 utc jmettraux don't hesitate to update the doc on the go : http://ruote.rubyforge.org/implementing_a_storage.html ;-)
2011-07-01 08:06:44 utc jmettraux ruote-kit ; https://github.com/kennethkalmer/ruote-kit/network I've noticed that forkers were motivated by haml versions and co
2011-07-01 08:07:09 utc tosch_le i don't implement a storage ;-)
2011-07-01 08:08:58 utc jmettraux FsWatchStorage
2011-07-01 08:09:03 utc jmettraux BayWatchStorage
2011-07-01 08:09:34 utc tosch_le ruote-kit: i think it's fine to use >= more often. since we have Gemfile.lock now, we can say that we've tested on version x.y.z when anything breaks
2011-07-01 08:10:22 utc tosch_le FsStorageWatcherWorker
2011-07-01 08:10:45 utc jmettraux +1
2011-07-01 08:10:59 utc jmettraux so it's a worker
2011-07-01 08:11:19 utc tosch_le sure it is. the storage is dumb, you remember? ;-)
2011-07-01 08:11:26 utc jmettraux that raises the question of : should we re-implement the worker thing
2011-07-01 08:11:49 utc tosch_le i'll create a subclass of Worker.
2011-07-01 08:11:51 utc jmettraux the worker is dumb currently : 1 polling thread
2011-07-01 08:12:48 utc tosch_le dumb was the wrong word. the storage is only the persistence layer, it doesn't process any messages or schedules.
2011-07-01 08:14:08 utc tosch_le i don't think it's necessary to re-implement the worker, it's fine the way it is. and you designed ruote 2.1 in a way that it's really easy to use another worker.
2011-07-01 08:14:52 utc jmettraux ok
2011-07-01 08:14:53 utc tosch_le the only thing that would be great was when i could run ruote's functional tests against the new worker instead of Ruote::Worker
2011-07-01 08:15:27 utc tosch_le like you can change the storage class to use
2011-07-01 08:16:10 utc jmettraux true
2011-07-01 08:18:42 utc tosch_le and i'd like to have access to the path of the storage directory fsstorage uses. right now, i'd have to use @storage.get_instance_variable(:cloche).get_instance_variable(:path)@
2011-07-01 08:19:55 utc jmettraux we can keep track of the cloche dir as an instance variable in FsStorage
2011-07-01 08:20:18 utc tosch_le that would be great.
2011-07-01 08:20:19 utc jmettraux or add a #path method to FsStorage that asks cloche for the path
2011-07-01 08:20:57 utc tosch_le anything like that would be appreciated.
2011-07-01 08:21:03 utc jmettraux let me add something quickly then
2011-07-01 08:21:17 utc tosch_le thanks!
2011-07-01 08:22:22 utc jmettraux https://github.com/jmettraux/ruote/commit/00a7ff0a4045e9ea20ecfc71a8ab98408286bfdd
2011-07-01 08:23:56 utc tosch_le thanks again.
2011-07-01 08:25:08 utc jmettraux you're welcome
2011-07-01 08:30:25 utc jmettraux I'd put the file watching functionality in a storage
2011-07-01 08:30:44 utc jmettraux because it's fs dependent
2011-07-01 08:31:09 utc jmettraux but that would still require a new worker
2011-07-01 08:31:26 utc jmettraux pull vs push
2011-07-01 08:31:46 utc tosch_le i'd raise an exception in the initializer: complain when the given storage isn't a fsstorage
2011-07-01 08:32:35 utc tosch_le it's a mixed thing; it's a worker that depends on a special storage
2011-07-01 08:32:37 utc jmettraux wait, we still want to read the storage when a worker is not present
2011-07-01 08:32:45 utc tosch_le well, not special, but well known
2011-07-01 08:32:58 utc jmettraux ok, makes sense
2011-07-01 08:33:18 utc jmettraux such a worker, I would need it for ruote-redis as well
2011-07-01 08:33:29 utc jmettraux even for ruote-couch
2011-07-01 08:33:55 utc tosch_le you'd need push functionality in redis or couchdb
2011-07-01 08:34:08 utc jmettraux it's already in there blpop
2011-07-01 08:34:10 utc jmettraux and co
2011-07-01 08:34:34 utc tosch_le ah, never dived that deep into redis. sounds amazing.
2011-07-01 08:35:25 utc jmettraux maybe if the storage presents a hook point to the worker, it could switch to listen instead of poll
2011-07-01 08:35:44 utc jmettraux sorry, I'm trying to scale your idea to all the storages
2011-07-01 08:38:28 utc jmettraux such a worker could be used by the 3 storages
2011-07-01 08:38:35 utc jmettraux the three stooges
2011-07-01 08:39:10 utc tosch_le it's not that easy with multi-worker-environments: to which worker shall the storage push?
2011-07-01 08:39:58 utc jmettraux all
2011-07-01 08:40:18 utc jmettraux it depends
2011-07-01 08:40:27 utc jmettraux with redis, blop gives to 1 consumer
2011-07-01 08:40:40 utc jmettraux couch would notify everybody (http long poll)
2011-07-01 08:41:14 utc jmettraux your fsevent system would notify all its workers
2011-07-01 08:42:19 utc tosch_le hmm. each storage may provide a worker backend. if there is none, the default (polling) backend is used.
2011-07-01 08:43:14 utc tosch_le the worker could be em-based by default for making it easy to write other backends
2011-07-01 08:43:23 utc jmettraux :^(
2011-07-01 08:43:42 utc tosch_le evented worker. but em as dependency is a no-go, i suppose.
2011-07-01 08:44:22 utc jmettraux em is great, but I try to avoid depending on it
2011-07-01 08:45:05 utc jmettraux your special fs storage extension could be em based
2011-07-01 08:45:35 utc jmettraux the new special worker would just place hooks into the special storages (if the hook points are provided)
2011-07-01 08:45:50 utc jmettraux no hook points, back to polling
2011-07-01 08:46:14 utc jmettraux you would be totally free to use EM in your storage impl
2011-07-01 08:46:35 utc jmettraux ACTION emits coffee
2011-07-01 08:46:53 utc tosch_le ACTION emits Vita Cola
2011-07-01 08:46:57 utc tosch_le ;-)
2011-07-01 08:47:57 utc jmettraux thanks !
2011-07-01 08:47:58 utc tosch_le hmm, the storage could reply to a "worker_class" method
2011-07-01 08:48:13 utc tosch_le or something like that.
2011-07-01 08:49:06 utc jmettraux strorage.respond_to?(:on_msg) ? storage.on_msg(self) : poll
2011-07-01 08:49:36 utc jmettraux "worker_class" sounds sovietic
2011-07-01 08:49:43 utc jmettraux ;-)
2011-07-01 08:50:35 utc tosch_le rofl
2011-07-01 08:51:16 utc tosch_le what would storage.on_msg do?
2011-07-01 08:51:45 utc jmettraux more like
2011-07-01 08:51:46 utc tosch_le it would push the message to the worker
2011-07-01 08:51:49 utc tosch_le ?
2011-07-01 08:51:55 utc jmettraux storage.on_msg do |msg|
2011-07-01 08:51:59 utc jmettraux process(msg)
2011-07-01 08:52:01 utc jmettraux end
2011-07-01 08:52:23 utc jmettraux it would register a block in the storage
2011-07-01 08:52:37 utc jmettraux the storage would call that block each time there is a new msg
2011-07-01 08:52:46 utc jmettraux something like that
2011-07-01 08:53:15 utc tosch_le that way you may tell the storage to push messages somewhere. the receiver side is missing
2011-07-01 08:53:44 utc jmettraux the owner of the block is the receiver
2011-07-01 08:56:54 utc jmettraux worker says "hey, you the storage, please notify me of new msgs, simply execute that block that I give you with the msg as argument"
2011-07-01 08:57:55 utc tosch_le but you'll need different implementations on worker side for each storage
2011-07-01 08:58:55 utc jmettraux not if the kinky details of file watching / couchdb long polling / redis blopooping are dealt with the storage
2011-07-01 08:59:50 utc jmettraux ok, I have to go for now
2011-07-01 09:00:12 utc jmettraux I can work on the worker if you want
2011-07-01 09:00:17 utc tosch_le thanks for the great discussion!
2011-07-01 09:00:24 utc jmettraux +1 excellent !
2011-07-01 09:00:33 utc jmettraux you could do the FsStorage++
2011-07-01 09:00:38 utc jmettraux with Em watching
2011-07-01 09:00:43 utc tosch_le :-)
2011-07-01 09:00:57 utc tosch_le thrilled to see your idea in code
2011-07-01 09:01:41 utc jmettraux thanks for the excellent idea (fsevent and co)