Friday, February 3, 2017

Learning in the open - mobprogramming

One of the best things about mob programming (#mobprogramming on twitter) is the way that the team shares learning.

Our team recently worked on a tool to remove unused AWS stacks, dedicating quite a few lunchtimes to the cause. We decided it would be a great opportunity to try some mob programming, as it was a project that we were all excited about.

In the process we

  • Learnt some emacs
  • Learnt about using the Serverless framework (AWS lambda under the hood) for the first time
  • Worked out how we were going to test our application
  • Familiarised ourselves with recursion and promises in javascript
  • Came up against deployment and runtime problems

While each of these could have been done solo, mob programming gave us the opportunity to:

  • Talk the problems through, coming up with better design
  • Get comfortable asking for explaination when we're unclear - showing that we don't know it all
  • Bring something new to the table that other people don't know - even if we've just found it ourselves
  • Teach others
  • Share the joy of seeing something working that we've worked on so closely together

One of the biggest benefits is individuals showing they don't know everything. Often as software developers we suffer from imposter syndrome, but just exposing that we're all learning can encourage a better culture where people are happy being vulnerable - where they feel they can ask questions without judgement.

Have you tried mob programming? What do you think are the benefits? Drawbacks?

Tuesday, January 31, 2017

Spinning down your cloud costs

One of the best things about the cloud is the ability to spin up new infrastructure for your dev environment. You can have a completely isolated environment for a new branch that you're working on, allowing you to test your changes in isolation.

This is fantastic as you can speed ahead on changes unencumbered - the only problem is that you'll leave a trail of cash sitting in your cloud providers pockets, which often doesn't work out so well for your business.

To attack our cost blowouts, we started a side project in our lunchtime to figure out how we can stop the hurt, and came up with Batman built using Serverless. Most of our infrastructure is managed in a single place (cloudformation) with a set of tags associated with each stack including the slice (or branch name), and the cloudformation stack keeps a created date & last update date. Utilizing this information Batman runs every night to cleanup stacks that are on non-master branches where the stack hasn't been updated for 7 days. This is usually enough time for us to have tested the branch and integrated it into the master stack.

We've also recently added a webhook trigger that deletes a stack when the branch is deleted in git which saves us up to 7 days of running costs.

We've stopped over 300 stacks in a month, and at a guesstimate of $1 per day running costs for a stack, the dollars quickly add up.

There are many ways to save costs, and we've found that for our journey into AWS, Batman has come about at the right time. Not only has it been interesting to explore serverless, our team got to do a lot of the work with some mob programming which was really fun (more to come).

Monday, February 15, 2016

Postgres search - overlapping arrays

Postgres Overlapping Arrays

We've got some spare food hanging around that we need to get rid of, but have to figure out which of our animals will eat it. Luckily we've recorded what each of them eat in our database! Given the three animals

{ 'species': 'cat', 'name': 'frank', 'eats': ['fish', 'biscuits', 'pudding'] }
{ 'species': 'dog', 'name': 'frankie', 'eats': ['fish', 'vegemite', 'cat'] }
{ 'species': 'snake', 'name': 'francine', 'eats': ['biscuits'] }

We want to find who will take the fish and pudding that we have sitting around. We'll get them out of their cage and bring them to the one spot to feed them. How can we figure this out from our data?

If the document was modelled to it's 3rd normal form in sql we could do some joins and figure it out without too much drama. To do this in Postgres with our jsonb storage a simple way to achieve the same outcome is to store the values we want to search on - eats in our case - is to store those values in their own column as an array (in addition to the jsonb column) and then use the Postgres array comparisons to do a search where the eats column contains any of ['fish', 'pudding']. What does this look like using Massive?`select * from animals
  where eats && $1`,
  ['fish', 'pudding'],
  (err, results) => {
    // do something interesting with the results

Note that you'll need to add the extra column manually and populate it when saving a document:

db.animals.insert({body: jsonDocument, eats: jsonDocument.eats}, (err, results) => {})

Postgres - partial text search

Postgres Partial Text Search

Given the three documents below, we want to find the animals with name like 'frank'

{ 'species': 'cat', 'name': 'frank' }
{ 'species': 'dog', 'name': 'frankie' }
{ 'species': 'snake', 'name': 'francine' }

If this were sql we would search name like %frank% we'd expect to get the first two results. Postgres & Massive gives us a way to search for equality in a document so:

db.animals.findDoc({name: 'frank'}, (err, res) => { // do things with results })

Would only return the first result. (Note that Massive creates an index for your jsonb document if you let it create the table).

To search for partial results we need to add a Trigram Index - this breaks up text into three character chunks and gives a weighting to the search term match, rather than giving us an exact match or nothing. For the example above, we would break out the name property into another column (Massive creates a 'search' column which you could populate, or just make your own) - at this point we lose some of the ability to use Massive findDoc functions, but can still use the where function.

Creating the record would now look like

db.saveDoc("animals", {body: jsonDocument, name:}, function(err,res){});

And to query:

db.animals.where("name like $1", ['%frank%''], (err, result) => { // do something with the result });

Postgres - a document database?

Storing large json documents in SQL servers is painful. You first need to map the object out to a relational structure, then deal with putting it back together when you're querying. In addition you probably need to use some ORM to store your objects. All of which makes me sad.

With Postgres, there is a great feature that lets you store your json object into a column, and do queries on the json natively in the database!

I'm using Massive.js for working with Postgres in node, which has first class support for jsonb storage and makes the query syntax for working with jsonb documents a bit nicer. It also lets you write your own SQL when you need to do some more custom queries.

If you don't want any other columns in your table (it'll give you id and created date for free), you can use the saveDoc function (more on the doc storage syntax here):

db.saveDoc("my_documents", jsonDocument, function(err,res) {
    //the table my_documents was created on the fly with id and created date
    //res is the new document with an ID created for you

If you want to store some more data in your table then you can:

  • Manually create the table yourself prior to using Massive
  • Let Massive create the initial table for you, then add columns as needed (Massive will also create the primary key and auto-incrementing function which is handy)

We'll look at some querying strategies in the next post (there is documentation in the json section of the github repo which is a great place to start)

Tuesday, August 18, 2015

Testing SQL Server Connections

Testing SQL Server Connections
I found this gem today trying to confirm if a windows server had access to a SQL instance (thinking I'd have to install something and do 10 restarts) - turns out you can create a .udl file which will bring up simple window to test your database connection!

Friday, June 22, 2012

One of the issues of infrequent releases.....

It's been a month since our last deployment, and we've got two features that's have been sitting in code for longer than that, and one that's been recently developed. One feature is 'hidden', the other isn't functional yet (a couple of external dependencies).

There are a couple of problems here:

  • Not getting feedback on features that are ready to be released
  • Keeping the information in your brain so that you can fix any bugs that arise when the feature finally gets released
  • Trying to remember all the little bits that need to be completed to make the feature ready to release
A lot of this has to do with the flow of development, and perceived issues with releasing small features often. But the one that has kept on creeping up on me in the last couple of weeks is the realisation I want to keep adding new features, little ones that I think will make a difference, but aren't really necessary or proven to be required. 

This mentality comes from a "we're not going to be able to push anything new for another month", so try to pack everything in. It means that the code isn't as lean as it should be. Instead of waiting for feedback, reacting and pushing a new release, we just assume what people are after. 

Have you found this to be an issue when you're doing infrequent releases?