| 2011-07-26 07:47:11 utc | kennethkalmer | righto |
| 2011-07-26 07:47:17 utc | kennethkalmer | afternoon ! |
| 2011-07-26 07:47:20 utc | kennethkalmer | coffees ? |
| 2011-07-26 07:53:17 utc | jmettraux | kennethkalmer: oh please ! |
| 2011-07-26 07:53:47 utc | kennethkalmer | ACTION emits espresso dopio |
| 2011-07-26 07:54:11 utc | jmettraux | ACTION intercepts |
| 2011-07-26 07:54:12 utc | jmettraux | thanks ! |
| 2011-07-26 07:54:49 utc | kennethkalmer | trying to trace the issue with this one ruote instance refusing to launch processes |
| 2011-07-26 07:55:00 utc | kennethkalmer | trying to avoid nuking all the storages and starting fresh |
| 2011-07-26 07:55:30 utc | jmettraux | maybe you could write some code that updates the locks |
| 2011-07-26 07:55:55 utc | kennethkalmer | well, I manually DEL-ed the locks |
| 2011-07-26 07:55:59 utc | kennethkalmer | didn't fix a thing |
| 2011-07-26 07:56:13 utc | kennethkalmer | it is only on this one server, production and dev systems are running 100% |
| 2011-07-26 07:56:19 utc | kennethkalmer | only staging screwing me around |
| 2011-07-26 07:56:27 utc | kennethkalmer | so I definitely have an edge case |
| 2011-07-26 07:59:05 utc | kennethkalmer | could it be the composite storage maybe |
| 2011-07-26 08:02:14 utc | jmettraux | the launch msg doesn't get written to redis ? |
| 2011-07-26 08:03:09 utc | kennethkalmer | looking at my debug logs it seems the WFID isn't getting generated |
| 2011-07-26 08:03:47 utc | jmettraux | only launch operations do fail ? Do other kind of operations (like cancel) fail ? |
| 2011-07-26 08:04:01 utc | kennethkalmer | nope, they work fine |
| 2011-07-26 08:04:13 utc | jmettraux | jammed wfid generator ? |
| 2011-07-26 08:04:56 utc | kennethkalmer | using the stock mnemo generator, now looking thru the source there to see if something could act up |
| 2011-07-26 08:05:56 utc | kennethkalmer | redis does have a launch attempt from earlier today |
| 2011-07-26 08:06:05 utc | kennethkalmer | get "ruote:iiab:staging:variables/last_wfid" |
| 2011-07-26 08:06:05 utc | kennethkalmer | "{\"_rev\":\"3\",\"put_at\":\"2011-07-26 05:57:56.067646 UTC\",\"_id\":\"last_wfid\",\"type\":\"variables\",\"raw\":1312156830.73028}" |
| 2011-07-26 08:06:29 utc | kennethkalmer | i just tried to launch a process through rk though, no luck |
| 2011-07-26 08:06:53 utc | kennethkalmer | the "put_at" should have reflect "2011-07-26 10:05" |
| 2011-07-26 08:08:06 utc | jmettraux | indeed |
| 2011-07-26 08:09:13 utc | kennethkalmer | worker is running, redis monitor shows the polling |
| 2011-07-26 08:17:51 utc | jmettraux | worker not running would prevent any action |
| 2011-07-26 08:18:16 utc | kennethkalmer | yeah, true, just thinking out aloud |
| 2011-07-26 08:18:25 utc | jmettraux | where does the launch stop (before the put_msg obviously) but where does it stop |
| 2011-07-26 08:19:03 utc | kennethkalmer | my best guess is @context.wfid.generate at this stage |
| 2011-07-26 08:19:35 utc | kennethkalmer | lib/ruote/receiver/base.rb |
| 2011-07-26 08:19:40 utc | kennethkalmer | 68 |
| 2011-07-26 08:20:27 utc | jmettraux | where in the generate method ? |
| 2011-07-26 08:20:52 utc | kennethkalmer | ACTION is going to add some traces now |
| 2011-07-26 08:21:03 utc | jmettraux | some puts in the mnemo_wfid_generator would help us :-) |
| 2011-07-26 08:27:44 utc | kennethkalmer | stops in "raw = get_raw" in mnemo_wfid_generator |
| 2011-07-26 08:27:50 utc | kennethkalmer | adding more logs now |
| 2011-07-26 08:31:28 utc | kennethkalmer | stops here |
| 2011-07-26 08:31:29 utc | kennethkalmer | raw = raw + 0.01 while raw.to_f <= lraw |
| 2011-07-26 08:31:38 utc | kennethkalmer | in get_raw() |
| 2011-07-26 08:32:05 utc | kennethkalmer | @last: {"_rev"=>"5", "put_at"=>"2011-07-26 08:25:59.546852 UTC", "_id"=>"last_wfid", "type"=>"variables", "raw"=>1312156830.75751} |
| 2011-07-26 08:32:11 utc | kennethkalmer | lraw: 1312156830.76751 |
| 2011-07-26 08:33:26 utc | kennethkalmer | aha |
| 2011-07-26 08:33:39 utc | kennethkalmer | Time.at(1312156830.76751) |
| 2011-07-26 08:33:41 utc | kennethkalmer | => 2011-08-01 02:00:30 +0200 |
| 2011-07-26 08:37:58 utc | kennethkalmer | ACTION brewing a celebratory cup |
| 2011-07-26 08:41:10 utc | jmettraux | well done ! |
| 2011-07-26 08:41:55 utc | jmettraux | there should be a way to work that around |
| 2011-07-26 08:42:06 utc | jmettraux | to make it not fail |
| 2011-07-26 08:57:18 utc | kennethkalmer | coffee is served -> http://dl.dropbox.com/u/509720/DSC_0108.jpg |
| 2011-07-26 08:57:55 utc | jmettraux | woah, nice cups !! |
| 2011-07-26 08:58:12 utc | kennethkalmer | +1 |
| 2011-07-26 08:58:17 utc | kennethkalmer | love them |
| 2011-07-26 08:58:46 utc | kennethkalmer | will send you some :D |
| 2011-07-26 09:00:08 utc | kennethkalmer | sidenote, it does seem to launch the processes |
| 2011-07-26 09:00:23 utc | kennethkalmer | the wfid's do get generated |
| 2011-07-26 09:00:24 utc | kennethkalmer | it just takes so long that the browsers timeout |
| 2011-07-26 09:01:11 utc | kennethkalmer | in these cases now it took 9 minutes to advance from the current time to a time past 1 Aug |
| 2011-07-26 09:02:26 utc | kennethkalmer | now that we know the cause, I'm gonna purge the storage |
| 2011-07-26 09:02:32 utc | kennethkalmer | it is staging after all |
| 2011-07-26 09:03:28 utc | kennethkalmer | oh, and thanks for writing the copy_to() on StorageBase |
| 2011-07-26 09:03:40 utc | kennethkalmer | was thinking that was going to be my next contribution :) |
| 2011-07-26 09:06:00 utc | jmettraux | ah ! |