| 2010-06-23 04:27:24 utc | mnichols | Hi all |
| 2010-06-23 04:29:40 utc | mnichols | Question for the collective... Is it normal to have 180mb in mu msgs dir? |
| 2010-06-23 04:30:43 utc | mnichols | I thought that was supposed to be kind of transient. ?? |
| 2010-06-23 04:33:21 utc | mnichols | I also have several message files that say something like "-bash: !23456248307460: event not found". Not sure what that means.?? |
| 2010-06-23 05:07:15 utc | jmettraux | hello |
| 2010-06-23 05:07:24 utc | jmettraux | msgs should be empty |
| 2010-06-23 05:07:45 utc | jmettraux | it's supposed to be consumed on the fly |
| 2010-06-23 05:08:05 utc | jmettraux | if you're sure that your engine is OK, you can wipe out that msgs dir |
| 2010-06-23 05:08:35 utc | jmettraux | for the "-bash !xxx" thing, do you have more details ? |
| 2010-06-23 05:31:26 utc | mnichols | Sorry. Didn't see your reply. |
| 2010-06-23 05:32:05 utc | mnichols | I just rebuilt my server and started up a worker and it looks like it is consuming the msgs. |
| 2010-06-23 05:32:33 utc | jmettraux | great |
| 2010-06-23 05:32:53 utc | mnichols | Those msg files with the -bash message... That's all it says. Is there something else I can look at? |
| 2010-06-23 05:34:44 utc | jmettraux | could you please gist one of those ? |
| 2010-06-23 05:35:08 utc | mnichols | k |
| 2010-06-23 05:36:27 utc | mnichols | http://gist.github.com/449552 |
| 2010-06-23 05:36:49 utc | jmettraux | woah |
| 2010-06-23 05:36:57 utc | mnichols | I have 322 similar messages. |
| 2010-06-23 05:37:01 utc | jmettraux | never seen such a symptom |
| 2010-06-23 05:37:05 utc | mnichols | (that I've found) |
| 2010-06-23 05:37:56 utc | jmettraux | ruote doesn't have any "event not found" string |
| 2010-06-23 05:38:10 utc | mnichols | Spooky |
| 2010-06-23 05:38:57 utc | mnichols | Might if have something to do with the cloche file locks? Maybe out of file descriptors??? |
| 2010-06-23 05:39:11 utc | mnichols | (stab in the dark) |
| 2010-06-23 05:39:14 utc | jmettraux | maybe |
| 2010-06-23 05:45:23 utc | mnichols | Wow... worker is really flying through these messages now. I think I may have had some systemic problems with old server. The "-bash" leads me to believe there was something wrong at the resource/system level. |
| 2010-06-23 05:45:59 utc | jmettraux | how many processes do you usually have ? |
| 2010-06-23 05:47:04 utc | mnichols | We go in large bursts. Right now I'm around 100k |
| 2010-06-23 05:47:12 utc | mnichols | Up to 200k |
| 2010-06-23 05:48:33 utc | jmettraux | wow |
| 2010-06-23 05:49:07 utc | mnichols | I have it on an EBS Volume. I could share a snapshot with you if it would be helpful to you. |
| 2010-06-23 05:49:36 utc | jmettraux | I'm afraid I wouldn't have to time to look at it :-( |
| 2010-06-23 05:50:02 utc | mnichols | np |
| 2010-06-23 05:50:32 utc | jmettraux | have the time |
| 2010-06-23 05:51:18 utc | mnichols | I'll keep an eye out for the issue again. Anything in particular you think I should monitor? |
| 2010-06-23 05:52:11 utc | jmettraux | I would need more context |
| 2010-06-23 05:52:30 utc | jmettraux | if you could give me instructions to reproduce the issue |
| 2010-06-23 05:52:50 utc | jmettraux | on how to reproduce the issue |
| 2010-06-23 05:53:21 utc | jmettraux | nobody else reported such an issue |
| 2010-06-23 05:54:01 utc | mnichols | I'm clueless right now. Just trying to get through this big job. |
| 2010-06-23 05:54:30 utc | mnichols | Where do message errors normally appear? |
| 2010-06-23 05:54:56 utc | jmettraux | next to /msgs/, under /errors/ |
| 2010-06-23 05:55:04 utc | mnichols | Seems like there used to be a directory that they went in, but it went away with 2.0 |
| 2010-06-23 05:55:08 utc | mnichols | oh. |
| 2010-06-23 05:55:34 utc | mnichols | I don't have an errors dir |
| 2010-06-23 05:55:50 utc | jmettraux | do you suspect errors ? |
| 2010-06-23 05:56:41 utc | mnichols | No. I've tested my wf pretty well. But it seems that this would have triggered and "error". |
| 2010-06-23 05:57:25 utc | mnichols | Besides definition problems, what other types of errors could show up? |
| 2010-06-23 05:58:02 utc | jmettraux | runtime errors |
| 2010-06-23 05:58:08 utc | jmettraux | participants gone missing |
| 2010-06-23 05:58:13 utc | jmettraux | disk full |
| 2010-06-23 05:58:16 utc | jmettraux | ... |
| 2010-06-23 05:58:21 utc | mnichols | ah |
| 2010-06-23 05:59:13 utc | mnichols | So if I were to set up a monitor, I should look at /msgs/errors for new files? |
| 2010-06-23 05:59:56 utc | jmettraux | at /errors |
| 2010-06-23 06:00:10 utc | mnichols | k |
| 2010-06-23 06:00:36 utc | jmettraux | or, in ruby, you could directly ask the engine |
| 2010-06-23 06:00:49 utc | mnichols | engine.errors ? |
| 2010-06-23 06:00:57 utc | jmettraux | p "ARGH" if engine.errors.size > 0 |
| 2010-06-23 06:01:01 utc | jmettraux | :) |
| 2010-06-23 06:01:08 utc | mnichols | ) |
| 2010-06-23 06:01:11 utc | mnichols | :) |
| 2010-06-23 06:01:16 utc | mnichols | Got it. |
| 2010-06-23 06:01:28 utc | mnichols | Different subject... |
| 2010-06-23 06:01:50 utc | mnichols | I tried that storage engine migration utility that you added to .10 |
| 2010-06-23 06:02:05 utc | mnichols | It worked awesome... until I ran our of memory. :) |
| 2010-06-23 06:02:45 utc | mnichols | Looks like get_many loads all of the items. It choked on expressions. |
| 2010-06-23 06:03:04 utc | jmettraux | aaah |
| 2010-06-23 06:03:06 utc | jmettraux | too bad |
| 2010-06-23 06:03:22 utc | jmettraux | if you could document that failure in the mailing list |
| 2010-06-23 06:03:25 utc | mnichols | Thought I could just split up the dirs and take a chunk at a time. |
| 2010-06-23 06:03:59 utc | jmettraux | that would make up for a good TODO |
| 2010-06-23 06:04:32 utc | mnichols | I looked into a way to make it load chunks at a time, but that would require mods to all of the storage engines. They all take their own opts={} |
| 2010-06-23 06:05:55 utc | mnichols | Not a high priority for me right now. I was hoping to get my expressions into couch so I could do some fancy querying. |
| 2010-06-23 06:07:09 utc | jmettraux | querying on workitems or expressions ? |
| 2010-06-23 06:09:32 utc | mnichols | I guess workitems |
| 2010-06-23 06:09:42 utc | mnichols | Want to see current status |
| 2010-06-23 06:12:02 utc | mnichols | I've tried querying the engine, but FsStorage takes a while to pull up a process when there are a lot. |
| 2010-06-23 06:12:27 utc | jmettraux | I should do something about that |
| 2010-06-23 06:12:51 utc | mnichols | Wouldn't couch be faster? |
| 2010-06-23 06:13:09 utc | mnichols | At least for lookup? |
| 2010-06-23 06:13:58 utc | jmettraux | probably |
| 2010-06-23 06:14:19 utc | jmettraux | but I should find a way to speed up FsStorage anyway |
| 2010-06-23 06:15:37 utc | mnichols | I saw your blog post saying you are using ruote-couch in a couple of projects. What do you like about it? |
| 2010-06-23 06:16:19 utc | jmettraux | I use couch for the rest of the data, so everything goes in couch |
| 2010-06-23 06:16:24 utc | jmettraux | but it's slow... |
| 2010-06-23 06:16:34 utc | mnichols | oh |
| 2010-06-23 06:16:57 utc | jmettraux | http://ruote.rubyforge.org/configuration.html#storage |
| 2010-06-23 06:17:02 utc | jmettraux | maybe lookup is fast |
| 2010-06-23 06:17:26 utc | jmettraux | but the "it goes over the network and back" "via http" is present |
| 2010-06-23 06:17:34 utc | mnichols | right |
| 2010-06-23 06:17:56 utc | mnichols | I saw this. Redis is an interesting option. |
| 2010-06-23 06:18:44 utc | mnichols | I'm not sure what size constraints it has though. (have it on my todo list to check into it) |
| 2010-06-23 06:20:53 utc | jmettraux | redis < 2.0 requires you to have everything in memory |
| 2010-06-23 06:21:07 utc | jmettraux | redis >= 2.0 requires you to have all the keys in memory |
| 2010-06-23 06:21:14 utc | mnichols | I suspected. |
| 2010-06-23 06:21:36 utc | jmettraux | maybe the last message in there : http://groups.google.com/group/openwferu-users/browse_frm/thread/5cb80730419a9a38 might interest you |
| 2010-06-23 06:22:29 utc | mnichols | Oh, yeah I saw this. One of the reasons I was investigating couch. |
| 2010-06-23 06:23:39 utc | mnichols | What is stored in "workitems"? |
| 2010-06-23 06:24:08 utc | jmettraux | workitems |
| 2010-06-23 06:24:16 utc | mnichols | :) |
| 2010-06-23 06:24:18 utc | jmettraux | well |
| 2010-06-23 06:24:21 utc | mnichols | Really? |
| 2010-06-23 06:24:35 utc | jmettraux | the once managed by Ruote::StorageParticipant participants |
| 2010-06-23 06:25:39 utc | jmettraux | the ones... |
| 2010-06-23 06:25:51 utc | mnichols | So why do I not have a workitems dir? |
| 2010-06-23 06:25:52 utc | jmettraux | (sorry writing in another language in other windows) |
| 2010-06-23 06:25:57 utc | mnichols | :) |
| 2010-06-23 06:25:58 utc | mnichols | np |
| 2010-06-23 06:26:10 utc | jmettraux | maybe you don't use a Ruote::StorageParticipant |
| 2010-06-23 06:26:48 utc | mnichols | Not required? |
| 2010-06-23 06:29:22 utc | mnichols | Where would workitems be stored if Ruote::StorageParticipant is not used? |
| 2010-06-23 06:30:00 utc | jmettraux | if you didn't register any Ruote::StorageParticipant no need for /workitems |
| 2010-06-23 06:30:15 utc | jmettraux | I guess you have participants that use them directly and so and do |
| 2010-06-23 06:30:17 utc | jmettraux | and so |
| 2010-06-23 07:07:54 utc | jmettraux | tosch_le: hello Torsten |
| 2010-06-23 07:08:06 utc | tosch_le | hello john! |
| 2010-06-23 07:08:41 utc | jmettraux | how are you doing ? |
| 2010-06-23 07:08:57 utc | tosch_le | fine, thanks. and you? |
| 2010-06-23 07:09:18 utc | jmettraux | doing fine :) |
| 2010-06-23 07:14:59 utc | mnichols | Thanks for you help, John. Talk to you later. |
| 2010-06-23 07:15:14 utc | jmettraux | mnichols: you're welcome ! |
| 2010-06-23 07:42:41 utc | lbt | o/ |
| 2010-06-23 16:23:15 utc | lbt | woohoo... a python participant :) |
| 2010-06-23 16:36:07 utc | kennethkalmer | lbt: awesome stuff ! |
| 2010-06-23 16:36:25 utc | kennethkalmer | just merge jmettraux & your work into my branch |
| 2010-06-23 16:36:28 utc | kennethkalmer | and push to github |
| 2010-06-23 16:36:38 utc | kennethkalmer | can I push a new ruote-amqp gem to gemcutter ? |
| 2010-06-23 17:18:21 utc | lbt | kennethkalmer: I think we need to check with john before cutting another gem |
| 2010-06-23 17:18:40 utc | lbt | I'd like to put the python code into r-a though |
| 2010-06-23 17:19:49 utc | kennethkalmer | np on checking with john |
| 2010-06-23 17:20:00 utc | kennethkalmer | i'm very skeptical on having python code in the repo |
| 2010-06-23 17:20:12 utc | lbt | why? :) |
| 2010-06-23 17:20:22 utc | lbt | it's ruote... not ruby :) |
| 2010-06-23 17:20:25 utc | lbt | ? |
| 2010-06-23 17:20:34 utc | kennethkalmer | it is ruby :) |
| 2010-06-23 17:20:38 utc | lbt | heh |
| 2010-06-23 17:20:51 utc | kennethkalmer | why not publish it as a separate repo/egg ? |
| 2010-06-23 17:21:02 utc | kennethkalmer | ACTION python-fu is very low |
| 2010-06-23 17:21:05 utc | lbt | well, we could... |
| 2010-06-23 17:21:19 utc | kennethkalmer | i think it would be best |
| 2010-06-23 17:21:23 utc | lbt | here: http://gist.github.com/450248 |
| 2010-06-23 17:21:40 utc | lbt | it's almost ruby |
| 2010-06-23 17:22:03 utc | lbt | except the comments come after the def and not before... |
| 2010-06-23 17:22:31 utc | kennethkalmer | :) |
| 2010-06-23 17:22:33 utc | lbt | s/end// <---- lbt's patented ruby2python converter |
| 2010-06-23 17:22:46 utc | kennethkalmer | i think a separate project is best |
| 2010-06-23 17:22:48 utc | lbt | OK |
| 2010-06-23 17:22:58 utc | kennethkalmer | thanks for understanding |
| 2010-06-23 17:23:06 utc | kennethkalmer | pretty cool though |
| 2010-06-23 17:23:10 utc | kennethkalmer | have to admit it kicks ass |
| 2010-06-23 17:24:30 utc | lbt | bbiab ... chicken+mango kebabs.... mmm |
| 2010-06-23 17:25:23 utc | kennethkalmer | enjoy |
| 2010-06-23 17:25:29 utc | kennethkalmer | i'm also starting dinner in a bit |
| 2010-06-23 18:41:45 utc | lbt | kennethkalmer: one thing I did find... r-a is using the anonymous exchange; ie "" |
| 2010-06-23 18:42:35 utc | lbt | thinking it should probably use "ruote-participants" or something |