x You must enable Federated Login Before for this application.
Google App Engine Control Panel -> Administration -> Application Settings -> Authentication Options

Blog Tiny Probe

Building a Startup Framework for AppEngine

Thu, 31 Oct 2013

by kordless

It's time I built a framework.

I code in Python and JavaScript and I'm decent at doing HTML layout and simple CSS design work. I'm familiar with JQuery and Bootstrap. I know how to use Webapp2 and Jinja. I was recently able to write an simple Oauth library for talking to Github.

AppEngine is a good choice for hosting infrastructure as it's fairly inexpensive, supports Python, includes methods for user authentication, and provides simple database storage. It's also exceedingly handy to have integration with a few other Google products including BigQuery and Compute Engine.

On the other hand, AppEngine requires a lot of work to implement federated logins, secure form handling, extensible templating, and tight integration with other services, like Github and Twitter.

In the past, I've used GAE-Boilerplate to implement several sites, including StackGeek, TinyProbe and Utter.io. Each time I do a new site, I think to myself how much easier it would be if only I had a slightly different framework at hand to quickly implement those sites. I started doing a small amount of contributing to GAE-Boilerplate last year, but became quickly disappointed with the lack of focus on core features for building features required by a startup site, and the ease of its setup.

Deciding to write a framework on AppEngine is one I'm doing primarily for personal reasons. Doing this will help me learn more about programming and hopefully give me the tools I need to quickly deploy and develop new ideas. A side effect is that it may also help other startups implement the first version of their product in a short amount of time.

All of this will be Open Source and available under the Pink Panthers project on Github. I'm naming after the Serbian crime syndicate, The Pink Panthers because, a) they are fearless badasses and b) they are clearly opinionated. Like me!

So, what does a startup need from a framework?

Let's take a look at a few features a software startup may need from a framework. BTW, when I use the term 'startup', I mean an early stage company, pre-product, with a few founders hammering out code. I'm definitely not referring to a later stage startup with B round funding, 1000 customers, and 120 employees! Those are not startups in my opinion.

An opinionated view of what features an early stage startup will need consists of the following:

  • a domain serving content
  • simple design for displaying content
  • simple way for non-technical people to create content
  • simple deployment methodologies, including testing
  • email registration for interested users
  • user registration for early beta testers
  • mailing list to email users periodically
  • forums to hold discussions
  • blogging framework for writing posts and making comments
  • social networking features
  • issue tracking via Github
  • source code versioning via Github
  • simple analytics dashboards
  • simple monitoring of the service
  • an A/B testing framework
  • an integrated payment system via Stripe

Installation

Installation, initial configuration, and creating a project on AppEngine will be done using a build script. A configuration webpage may be used for speeding up the installation. See Wordpress.

Installation documentation will be provided where scripts or GUIs cannot provide assistance in setting up the framework.

Installation should be able to be done by someone who has a marginal amount of technical expertise. It should be approachable by end developers who can do simple HTML editing and basic command line actions like checking out repositories via Git and editing content online at Github.

User Features

Features for managing users will be a priority. A startup depends on users for it's eventual success, and those users need ways of communicating with the startup and promoting the startup's offerings to other users. This means features such as blog posts, mailing lists, social media integration, commenting, analytics and A/B testing are available on the site from day one.

Code Layout

The layout and content of the framework should be made obvious with a sample site repo which others can clone and easily edit or repurpose within the limits of the underlying frameworks.

The content of the sample repo will drive the homepage page for Pink Panthers and the page it builds will contain blog posts and documentation for using the framework. The code for Pink Panthers will live inside the startup project's directory as a git module.

Dependencies which are being actively developed in their own git repos will live in git modules under the Pink Panther's repo. Dependencies which aren't updated regularly, or require custom modifications to work in the framework, will be cloned and added to the Pink Panther's repo.

Storage and Security

Content should be stored independently of any other service, with the exception of the startup's own Google account services and Stripe. Company critical data like user account information should be kept secure. Credentials or sensitive data like credit cards should have secure and simple methods for storage and processing.

Methods for backing up or exporting data should be readily available within the framework.

The code for the framework is Open Source, which allows it to be independently audited.

What else?

If you have any suggestions, it's time to make them in the comments below. I'll update this post as needed over the next few weeks time.

0 Comments

The Real Data Scientists: Developers

Thu, 29 Nov 2012

by kordless

I've been noticing the term data scientist being bandied about and loosely coupled to the term analyst. I think it brings a little zing to older dogeared terms like big data and cloud computing! Watch...

Friend: What do you do for a living?

Me: I'm a frickin data scientist, yo!

Impressive.

Actually, I'm Just a Programmer

If I remember correctly, my college degree's title was Computer Science/Mathematics. I suppose it's fair to state my degree technically makes me a computer scientist. To ride that logic train a bit further, as a programmer I can't really get away from dealing with data, so all of us developers just become data scientists don't we?

Analysts? Well, I suppose they know what questions to ask, but it's unlikely they know how to code. That begs the question: Which is harder, teaching an analyst to code, or teaching a coder to analyst? Analyze? Whatever.

Frankly, seeing how a lot of developers end up being tasked with writing dashboards and admin panels for their company's web applications, I think it's OK we start calling out these developers as the real scientific heros. After all, physicists regularly take up cellular biology as a career, because they are uniquely qualified to become experts with specific bio-technologies. Why can't computer scientists do the same with business logic?

They can and they do.

Application Monitoring vs. Application Intelligence

Over the last 5 years of my foray into operation management software, I noticed a trend while talking to customers about monitoring their applications: They wanted to know more about how users used their software using the same data they were using for monitoring. I know, earth shattering stuff, right? It seems obvious, but why exactly were they having that conversation with me when we started out talking about monitoring log files?

The answer lies in what role I was in at the time and to whom I was talking. As a director of the developers program at Splunk and more recently as a founder/CEO of Loggly, the people I talked to first were usually product/project managers. As the conversations progressed, they invariably fell to application intelligence, because these managers also worried about whether or not people used their software and how.

If you are collecting a ton of data for monitoring your app, it's not a stretch to extend it to collecting data about signups, features usage patterns or any other random metric you might dream up while sitting awake at 2AM worrying about your conversions. Why the hell wouldn't you track this stuff, given you had a tool that would do it?

The answer is, you would. All you need is a developer.

Apps to the Rescue! Not.

SalesForce was one of our first really BIG customers at Splunk. As many of you are probably aware, the awesome thing Splunk has going is its backend. It's essentially a non-relational key-value store shoehorned into a massively scalable search engine. This ability to scale and search a massive amount of logs is what originally got SalesForce's attention. What kept them interested and eventually made them purchase Splunk, was the ability to write apps to query that engine and display graphs to their customers.

Human Earf

I ended up working with a handful of developers assigned to the Splunk project at SalesForce. Over the course of a month or so, I helped those developers write a few of the early prototypes for showing mail campaign reports/graphs to customers using SalesForce. Basically the idea was you'd email a bunch of your company contacts with some offer and then SalesForce would show you how many of those emails bounced. You could then search (using Splunk) for the records that bounced and update your contact entries as needed.

It was a simple idea, but one that got us thinking: Was it possible to write a bunch of useful apps like this and sell them to others? At the time, a lot of us at Splunk sure thought so - that's basically what SplunkBase is all about, after all.

After doing Loggly and thinking about it a bit more with a ton of other customers, I'm not so sure it makes a ton of sense.

Keep IT Simple

The primary problem with a one-size-fits-all application intelligence app is the data ingested needs to all be uniform. And when I say uniform, I mean uniform data coming out of your webapp and my webapp. The data flows differ.

If I signup customers using a simplified flow and you don't, how do I write an app that tracks conversions accurately for both of us? The short answer is you'll just end up using an app to track whether someone signups up or not. And to hell with any data gleaned in between the first page load and the signup.

And there's the problem. It's just stupid: throwing away all that valuable user data.

Hey Mom! WTF is up with this peg?

But we do it everyday. With Google Analytics, MixPanel, ChartBeat, Geckoboard, and countless other application tracking sites who claim to offer detailed analysis of your user's activities. In reality all these applications offer is a low friction way for marketing peeps to glean some data from the torrent of user data your site gets everyday. Those people aren't data scientists. Or programmers.

And they are leaving a ton of questions about your business on the table.

With TinyProbe, I aim to change the way we approach data science by empowering the people who implement analytics solutions: developers. If you are a developer hacking on analytics for your applications, stick around. I think TinyProbe going to blow you away.

Want more info? Follow @TinyProbe, @StackGeek and @kordless on Twitter and get an invite now while you still can.

0 Comments