ruote log_2010-05-04

2010-05-04 01:35:56 utc jmettraux threetee: hello, I'm in the middle of replying to your email
2010-05-04 01:40:09 utc jmettraux replied
2010-05-04 01:45:37 utc threetee patching now
2010-05-04 01:45:58 utc jmettraux pinging Kenneth directly about it
2010-05-04 01:48:31 utc threetee seems to be working so far
2010-05-04 01:48:32 utc threetee :)
2010-05-04 01:48:39 utc jmettraux excellent
2010-05-04 01:59:48 utc threetee yes, everything seems to be working fantastic
2010-05-04 01:59:56 utc threetee thanks again for your help!
2010-05-04 02:00:02 utc jmettraux thanks for kicking the nest
2010-05-04 02:00:06 utc threetee heh, no problem
2010-05-04 02:00:26 utc threetee I'm looking at using this for a distributed systems management application at my work
2010-05-04 02:00:42 utc jmettraux oh cool
2010-05-04 02:01:07 utc jmettraux we'll do our best to help you
2010-05-04 02:02:41 utc threetee thanks, I'll be playing with this lots more tonight
2010-05-04 02:02:57 utc threetee I'll reply to the group once I have some updated POC code
2010-05-04 02:03:15 utc jmettraux much appreciated
2010-05-04 02:04:37 utc threetee incidentally, do you know of any good reference code I can look at for ruote 2.1 implementation? AMQP or not, anything would be helpful. I posted a similar question in the daemon-kit group, since the daemon-kit ruote template has a reference to some old openwfe code (http://groups.google.com/group/daemon-kit/browse_thread/thread/ef668ff090382257#)
2010-05-04 02:05:37 utc jmettraux I have some proprietary code
2010-05-04 02:05:42 utc jmettraux unfortunately
2010-05-04 02:05:54 utc threetee bummer
2010-05-04 02:06:07 utc jmettraux what kind of example do you need ?
2010-05-04 02:06:40 utc threetee my initial plan is to build a POC using ruote to manage a relatively simple process, like creating email accounts
2010-05-04 02:07:08 utc threetee because I work for a company with lots of lawyers, creating accounts on company machines involves a few approval steps, in addition to the actual work
2010-05-04 02:07:16 utc threetee automating and tracking the actual account creation is easy
2010-05-04 02:07:34 utc threetee but tracking the workflow through the various approval steps is hard to automate
2010-05-04 02:07:56 utc threetee so that's where I'm considering bringing in ruote
2010-05-04 02:08:04 utc jmettraux usually when people get stuck, are faced with a dilemma, they ask on the mailing list and I try to help
2010-05-04 02:08:23 utc threetee I thought about that, but that sounded too much like getting someone else to do my work for me :)
2010-05-04 02:08:26 utc jmettraux they come with half a process definition and I help sort it out
2010-05-04 02:08:40 utc threetee sounds good then
2010-05-04 02:09:07 utc jmettraux what kind of interface do you want to build ? web app ? rails ?
2010-05-04 02:09:14 utc threetee I'll work out the process definition and get the code working as much as I can, then I'll run it by the group to see where I went wrong :)
2010-05-04 02:09:28 utc threetee I already have a rails app that handles asset and metadata management for our systems
2010-05-04 02:09:55 utc threetee there's also an agent that lives on the systems and feeds data to the rails app
2010-05-04 02:10:16 utc threetee right now the data flows only from the agent to the management system
2010-05-04 02:11:09 utc threetee but I'll probably have the rails app handle the systems management duties as well, and have configuration instructions start flowing from it to the agents
2010-05-04 02:12:01 utc threetee actually, there may be a couple of apps that will provide a web interface for systems management
2010-05-04 02:12:13 utc threetee the current rails app is an internal tool, so that will probably stay internal
2010-05-04 02:12:23 utc jmettraux do you plan on having an standalone ruote engine ?
2010-05-04 02:12:24 utc threetee I also plan on adding a customer-facing tool as well
2010-05-04 02:12:28 utc threetee yes
2010-05-04 02:12:33 utc jmettraux great
2010-05-04 02:12:44 utc threetee because of the multiple apps, I figured I would need one
2010-05-04 02:14:17 utc threetee are there any non-proprietary code or use cases that you could share or point me to?
2010-05-04 02:14:45 utc jmettraux well, you've seen daustin's developments
2010-05-04 02:15:27 utc jmettraux my projects are much dumber, 1 engine, 1 worker, 1 storage, nothing extraordinary
2010-05-04 02:16:15 utc jmettraux there isn't much to show
2010-05-04 02:16:44 utc jmettraux 2 or 3 process definitions, a few tricks to show differents forms/screens based on the value of the fields of the workitems
2010-05-04 02:17:10 utc jmettraux if you tell me what you want to see, I can almost build it on the fly
2010-05-04 02:17:21 utc threetee I think barley is one of those examples, yes?
2010-05-04 02:17:34 utc jmettraux yes, but it's extremely bare
2010-05-04 02:17:39 utc threetee hence the name
2010-05-04 02:17:43 utc jmettraux :)
2010-05-04 02:18:10 utc jmettraux you'll probably have to choose a storage, set up a worker around it
2010-05-04 02:18:23 utc jmettraux and then, from your apps, instantiate an engine and point it to the storage
2010-05-04 02:18:50 utc jmettraux the work gets done in the worker (and in your amqp participants)
2010-05-04 02:19:07 utc jmettraux and the order / queries come from the engine (located in the front)
2010-05-04 02:19:29 utc jmettraux of course, once you have that you can make sure that orders come from other places than the front
2010-05-04 02:20:56 utc threetee cool, I'm not too far off in how I was planning to do things then
2010-05-04 02:21:20 utc threetee I gotta run, but I'll put some generic workflows together and post them. I'll also try to put together some docs and diagrams that describe how I envision the system working. I think I understand all the concepts, but I want to make sure that how I plan to put them into practice actually 1) will work, and 2) is the best way to do things
2010-05-04 02:21:44 utc jmettraux perfect
2010-05-04 02:21:53 utc threetee thanks again for the help!
2010-05-04 02:22:05 utc jmettraux you're welcome, thanks to you too
2010-05-04 02:22:15 utc threetee np
2010-05-04 17:45:16 utc kennethkalmer hi threetee !
2010-05-04 17:45:28 utc threetee hi kenneth
2010-05-04 17:46:35 utc threetee so about 8PM
2010-05-04 17:46:38 utc threetee nice
2010-05-04 17:46:40 utc kennethkalmer spot on
2010-05-04 17:47:48 utc threetee thanks for daemon-kit btw, it's very useful
2010-05-04 17:47:53 utc kennethkalmer it is a pleasure
2010-05-04 17:48:07 utc kennethkalmer please accept my sincere apologies for being so passive when you needed me
2010-05-04 17:48:17 utc kennethkalmer only starting to balance out my work now
2010-05-04 17:48:23 utc kennethkalmer been a rough year so far
2010-05-04 17:48:31 utc threetee np
2010-05-04 17:48:41 utc threetee my day job keeps me busy as well
2010-05-04 17:48:43 utc kennethkalmer will it be ok if i'm back in 2/3 hours ?
2010-05-04 17:48:53 utc kennethkalmer i need to clean & cook before i die of hunger
2010-05-04 17:48:56 utc threetee not at all
2010-05-04 17:48:59 utc kennethkalmer awesome
2010-05-04 17:49:02 utc kennethkalmer please hang around
2010-05-04 17:49:08 utc kennethkalmer i'll ping you when i'm back
2010-05-04 17:49:10 utc threetee I should be here
2010-05-04 17:49:13 utc threetee meeting schedule is light today :)
2010-05-04 17:49:21 utc kennethkalmer then we sort out this mess
2010-05-04 17:49:25 utc kennethkalmer good to here !
2010-05-04 17:49:37 utc kennethkalmer ttyl !
2010-05-04 20:24:36 utc kennethkalmer righto threetee
2010-05-04 20:24:46 utc kennethkalmer i'm back, checking out daustin's work
2010-05-04 20:26:33 utc threetee cool, I'm here as well
2010-05-04 20:28:52 utc kennethkalmer checking out the legacy instructions you mentioned as well, need to update that too
2010-05-04 20:31:19 utc threetee thanks
2010-05-04 20:31:39 utc threetee I'm putting together a quick diagram describing what I'm trying to accomplish for a quick POC
2010-05-04 20:36:22 utc kennethkalmer i'm building a quick test
2010-05-04 20:38:14 utc threetee here's an activity diagram I threw together to illustrate what I'm trying to accomplish: http://imgur.com/iDlIG
2010-05-04 20:38:54 utc threetee I don't do a lot of UML 2.0, so bear with me if there are glaring errors
2010-05-04 20:39:28 utc threetee anyway, swimlanes correspond to participants in the system
2010-05-04 20:39:38 utc threetee requestor, approver and form validator are all people
2010-05-04 20:40:13 utc threetee ingress mail servers and mailbox servers could represent any number of servers
2010-05-04 20:41:05 utc threetee I haven't yet drawn in other outputs from the system, like emails to requestor on success/failure, emails to approvers to notify of pending approvals, etc.
2010-05-04 20:42:42 utc threetee And I also haven't yet drawn in error conditions and what to do with them. For example, state needs to be consistent across all ingress mail servers, so configuration isn't complete until all have the same data
2010-05-04 20:44:10 utc threetee Oh and now that I look at this again, steps can be taken simultaneously on ingress mail servers and mailbox servers. However, there may be something to be said for having them be performed serially so you don't have to unwind as many changes if something goes screwy on the ingress server configuration step
2010-05-04 20:45:16 utc threetee anyway, that's a rough idea about what I'm trying to put together here, and ruote seemed like a logical choice to manage all of the steps
2010-05-04 20:46:04 utc threetee AMQP seems to make sense for communications between the ruote management service and the various nodes that need configuration, but I wouldn't mind input on whether or not this is actually the case
2010-05-04 20:52:36 utc kennethkalmer righto
2010-05-04 20:52:45 utc kennethkalmer i don't know UML for the smalls in a news paper
2010-05-04 20:52:47 utc kennethkalmer however
2010-05-04 20:52:56 utc kennethkalmer i do exactly the same thing as you do
2010-05-04 20:53:02 utc kennethkalmer with less people and more servers
2010-05-04 20:53:33 utc kennethkalmer and you have a good plan
2010-05-04 20:53:42 utc kennethkalmer i do the same with ruote-amqp + dk
2010-05-04 20:53:48 utc kennethkalmer that is mostly what i used it for
2010-05-04 20:53:55 utc kennethkalmer one question for you
2010-05-04 20:53:57 utc kennethkalmer just one
2010-05-04 20:54:04 utc threetee haha only one?
2010-05-04 20:54:08 utc kennethkalmer yeah :)
2010-05-04 20:54:29 utc kennethkalmer will the amqp communication be passing through a firewall, or wil it be secure behind the firewall ?
2010-05-04 20:54:54 utc threetee haha your question is along the lines of one of my quesitons
2010-05-04 20:55:40 utc kennethkalmer well, from experience AMQP through a firewall can be a bitch because of the reconnection issues when the fw drops idle connections
2010-05-04 20:55:50 utc kennethkalmer AMQP doesn't have a 'ping' thing going on like XMPP
2010-05-04 20:56:01 utc threetee ah, good to know
2010-05-04 20:56:32 utc kennethkalmer my ruote runs in ZA
2010-05-04 20:56:40 utc kennethkalmer with several agents running locally two
2010-05-04 20:56:46 utc kennethkalmer and a few run on EC2 and in the UK
2010-05-04 20:57:14 utc threetee I have four environments, three of which are spread throughout the US, one in China, and also EC2 servers on the east and west coasts
2010-05-04 20:57:29 utc kennethkalmer so
2010-05-04 20:57:32 utc kennethkalmer here is my advice
2010-05-04 20:57:34 utc threetee so yes, depending on how I architect the AMQP infrastructure, there is a very good chance that it could go through a firewall :)
2010-05-04 20:57:50 utc kennethkalmer first: have the daemon's send a heartbeat on another queue
2010-05-04 20:58:03 utc kennethkalmer that will help the connection stay alive
2010-05-04 20:58:19 utc kennethkalmer second, i'll be making a ruote-xmpp soon and bundling it with dk too
2010-05-04 20:58:25 utc kennethkalmer that will help a lot with firewall issues
2010-05-04 20:58:30 utc kennethkalmer since XMPP is very chatty
2010-05-04 20:58:33 utc threetee ah very nice
2010-05-04 20:58:34 utc kennethkalmer (no pun intended)
2010-05-04 20:58:40 utc threetee heh
2010-05-04 20:58:50 utc threetee I think nanite implements a heartbeat over AMQP, correct?
2010-05-04 20:59:22 utc kennethkalmer from what I recall yes
2010-05-04 20:59:37 utc kennethkalmer but that heartbeat is more of a need to know which agents are up
2010-05-04 20:59:44 utc kennethkalmer http://www.opensourcery.co.za/2009/04/19/to-amqp-or-to-xmpp-that-is-the-question/
2010-05-04 21:02:22 utc threetee good article, reading it now...XMPP is somewhat appealing because I suspect that a lot of my communications with agents will be one-to-one
2010-05-04 21:02:39 utc kennethkalmer mine is as well
2010-05-04 21:02:48 utc kennethkalmer but for use with ruote, it really doesn't matter
2010-05-04 21:02:56 utc threetee I'm not doing map/reduce stuff here, so I don't have a lot of need to throw messages out there for any node to handle
2010-05-04 21:02:56 utc kennethkalmer on the dk side it is a mere config change
2010-05-04 21:04:19 utc kennethkalmer i use queue names that match the fqdn's of the host running the daemon
2010-05-04 21:04:23 utc kennethkalmer has exact same effect
2010-05-04 21:04:28 utc threetee yes, that was my plan
2010-05-04 21:04:51 utc kennethkalmer seems great minds think alike ;)
2010-05-04 21:05:11 utc threetee heh, of course :)
2010-05-04 21:05:22 utc threetee I track other metadata about my servers, such as cluster membership, function, applications running
2010-05-04 21:05:46 utc threetee so I was planning to have fanout exchanges for each cluster, function, and other grouping
2010-05-04 21:06:24 utc threetee the servers know what groups they belong to, so they would subscribe to the appropriate exchanges, then I could send messages to groups as well
2010-05-04 21:09:44 utc kennethkalmer then up it one, and integrate chef-solo into the mix
2010-05-04 21:09:50 utc threetee I already use puppet :)
2010-05-04 21:09:54 utc kennethkalmer ah
2010-05-04 21:09:58 utc kennethkalmer i'm a cook ;)
2010-05-04 21:10:15 utc threetee and I'm a puppeteer
2010-05-04 21:14:16 utc kennethkalmer devops ftw
2010-05-04 21:17:07 utc threetee so from the blog post you pasted, it seems like you were pretty gung-ho about AMQP
2010-05-04 21:17:26 utc threetee but from what you're saying now, it seems like you use XMPP more
2010-05-04 21:17:42 utc threetee which one are you using more now?
2010-05-04 21:17:53 utc kennethkalmer AMQP still
2010-05-04 21:17:54 utc kennethkalmer love it
2010-05-04 21:18:04 utc kennethkalmer xmpp just crosses wan's better
2010-05-04 21:18:07 utc kennethkalmer i'm opting for a mix
2010-05-04 21:18:13 utc kennethkalmer lan == amqp
2010-05-04 21:18:17 utc kennethkalmer wan == xmpp
2010-05-04 21:18:35 utc kennethkalmer i just need a bit of json to go from A to B and back again
2010-05-04 21:18:42 utc kennethkalmer the transport doesn't bother me too much
2010-05-04 21:18:54 utc kennethkalmer 99% of the time they are both exactly the same
2010-05-04 21:19:01 utc threetee that idea had crossed my mind in the context of AMQP...I was considering setting up a broker for each environment
2010-05-04 21:19:05 utc kennethkalmer permitted you use an evented library
2010-05-04 21:19:18 utc kennethkalmer well
2010-05-04 21:19:26 utc kennethkalmer that could work
2010-05-04 21:19:33 utc threetee as long as my messages could be targeted
2010-05-04 21:19:49 utc threetee it would be difficult to send messages to groups that spanned multiple environments
2010-05-04 21:22:33 utc kennethkalmer well
2010-05-04 21:22:37 utc kennethkalmer amqp can do that
2010-05-04 21:22:41 utc kennethkalmer just trying to think now
2010-05-04 21:22:51 utc kennethkalmer there is a server for having brokers subscribe to each other
2010-05-04 21:22:56 utc kennethkalmer s/server/service/
2010-05-04 21:23:03 utc kennethkalmer haven't tried it yet
2010-05-04 21:25:37 utc threetee that's an interesting approach
2010-05-04 21:25:55 utc threetee I also thought about having one big AMQP cluster that spanned multiple environments, but I wasn't sure how feasible that was
2010-05-04 21:28:32 utc threetee On a slightly different topic, how are you securing your AMQP/XMPP communications? I know you can use SSL with AMQP (haven't looked into XMPP enough to know what it supports for encryption). However, I was concerned about what might happen if someone manages to guess the password to a queue and starts passing messages to my servers
2010-05-04 21:28:43 utc threetee obviously my AMQP infrastructure wouldn't be open to the world
2010-05-04 21:29:00 utc threetee but the majority of your threats come from within, right?
2010-05-04 21:29:22 utc threetee I was thinking some kind of PKI implementation so that communications from my management server could be signed
2010-05-04 21:30:17 utc kennethkalmer i have no idea truthfully
2010-05-04 21:30:28 utc kennethkalmer my google-fu is letting me down trying to find the broker to broker stuff
2010-05-04 21:30:56 utc threetee I'll make a note to check that out later
2010-05-04 21:31:11 utc threetee what are your thoughts about security?
2010-05-04 21:31:34 utc kennethkalmer in terms of amqp vs xmpp ?
2010-05-04 21:32:58 utc threetee that, and how to make sure that the messages your agents are acting on are legitimate
2010-05-04 21:33:13 utc kennethkalmer two good questions
2010-05-04 21:33:21 utc kennethkalmer two humiliating answers
2010-05-04 21:33:29 utc kennethkalmer first one is firewalling
2010-05-04 21:33:37 utc kennethkalmer oh, and ssl
2010-05-04 21:33:54 utc threetee right, I'll definitely be encrypting AMQP traffic
2010-05-04 21:34:07 utc kennethkalmer firewall allows me to make sure only authorized hosts could connect to the broker, but that isn't bullet proof
2010-05-04 21:34:16 utc kennethkalmer as for authenticity, i have nothing yet
2010-05-04 21:34:17 utc threetee right, somebody could spoof an IP
2010-05-04 21:34:39 utc kennethkalmer or they could have compromised a box and inject shit into the broker, you never know
2010-05-04 21:34:46 utc threetee yeah
2010-05-04 21:34:48 utc kennethkalmer nanite has an approach to sign messages
2010-05-04 21:34:53 utc threetee right, I saw that
2010-05-04 21:35:00 utc kennethkalmer haven't inspected it all yet
2010-05-04 21:35:15 utc kennethkalmer but you have a good point
2010-05-04 21:35:32 utc kennethkalmer should actually think about making a ruote-crypt or something
2010-05-04 21:35:48 utc kennethkalmer that any participant can use to implement encrypted/signed workitems
2010-05-04 21:36:05 utc threetee yeah, that would be pretty sweet
2010-05-04 21:48:54 utc kennethkalmer hmmph
2010-05-04 21:48:58 utc kennethkalmer testing woes
2010-05-04 21:50:48 utc kennethkalmer argh
2010-05-04 21:50:50 utc kennethkalmer idiot
2010-05-04 21:50:57 utc kennethkalmer been testing against ruote 2.1.9
2010-05-04 21:51:00 utc kennethkalmer need 2.1.10
2010-05-04 22:01:30 utc threetee doh!
2010-05-04 22:06:38 utc kennethkalmer hmm
2010-05-04 22:06:39 utc kennethkalmer working
2010-05-04 22:06:43 utc kennethkalmer quickstart as well
2010-05-04 22:06:55 utc kennethkalmer just realized an important thing here that I might need to update in dk still
2010-05-04 22:22:55 utc kennethkalmer eric
2010-05-04 22:23:05 utc kennethkalmer you happy with the dk & ruote-amqp changes ?
2010-05-04 22:23:17 utc kennethkalmer seems ok to me
2010-05-04 22:25:05 utc threetee yes, seems to be working fine
2010-05-04 22:25:19 utc kennethkalmer great
2010-05-04 22:25:21 utc threetee I'm working on building my ruote engine to match the diagram I sent
2010-05-04 22:25:50 utc threetee I'll let you and john know if I run into any more problems
2010-05-04 22:25:57 utc kennethkalmer i'll ping the list, have one question for john before i release ruote-amqp-2.1.6 & daemon-kit-0.1.8rc3
2010-05-04 22:25:59 utc kennethkalmer great
2010-05-04 22:26:13 utc threetee thanks for all your help, you guys have been great
2010-05-04 22:26:21 utc kennethkalmer no worries
2010-05-04 22:26:26 utc kennethkalmer we love to help
2010-05-04 22:26:29 utc threetee I'll come back with another round of questions, I'm sure :)
2010-05-04 22:26:40 utc kennethkalmer you're welcome anytime here
2010-05-04 22:26:48 utc threetee thanks
2010-05-04 22:26:58 utc kennethkalmer ask wayneeseguin, he hasn't asked a ruote question in months and still hangs around :)
2010-05-04 22:27:01 utc kennethkalmer yeah
2010-05-04 22:42:27 utc kennethkalmer hang around, we still have plenty of stories to share
2010-05-04 22:42:40 utc threetee sounds good, thanks again
2010-05-04 22:42:41 utc kennethkalmer and good luck with the testing
2010-05-04 22:42:47 utc kennethkalmer no worries
2010-05-04 22:42:51 utc kennethkalmer welcome aboard :D
2010-05-04 22:43:10 utc kennethkalmer ttyl
2010-05-04 23:20:23 utc brain_in_vat is it possible to refer to the config hash in the participant ref?
2010-05-04 23:20:28 utc brain_in_vat I basically want to do something like ...
2010-05-04 23:21:06 utc brain_in_vat cancel_process :if => "$config['workitems]['urls'].empty?"
2010-05-04 23:32:10 utc brain_in_vat nm, this ended up working
2010-05-04 23:32:10 utc brain_in_vat cancel_process :if => '${f:dataset_ids.size} == 0'