First talk was Randal Thomas from Engine Yard on ETL (Extrac, Transform, Load) applications that all of us have been asked to write in the past. These are those ‘quick, simple and one-use’ data parsing apps your accounting or HR or Business Analyst’s having been asking you about. His point is that they are never quick, never simple and you end up running them every day for years until they are rewritten.
My personal experience with an ETL was from a telecom company I worked for that over night expanded from a rebrander to a provider. The million dollar software they bought to handled the per call billing didn’t have any way to parse the huge daily call logs coming in from carriers. Thats what we call Enterprise! So I whipped up a quick 40 lines of Perl code and some Bash duct tape that became the single interface to batch load the milliions of records worth hundreds of thousands of dollars daily. And I was the Network Engineer at the time; the programmers were still working on an EJB config file to model the framework to set the display for an entry screen that would stub out the function that eventually would parse the call log.
Don’t get me wrong, Enterprise is not a four letter word (its clearly has ten letters) and Randall’s talk was on building ETLs grounded in reality. First, be lazy. Use google to see if someone already has done the hard work. Second, realize that users will lie to you. Expect to verify everything they tell you and don’t be surprised when it changes. Business Rules are really business generalizations, and unconstant ones at that. Use pipes when possible, chaining outputs from one application into inputs of another. Expect to have to stop and start a process in the middle, and allow for that. Learn SQL because there is no such thing as a perfect abstraction. Also learn from Map Reduce and use it when needed