Metacortex Ingestion API

With the recent announcement of Metacortex, I felt it prudent to share some of the documentation.

The coolest part about the system is that there are only three steps:

  1. Deploy Ingestion Server, Search Server, PostgreSQL backend
  2. Send comments to ingestion endpoint
  3. Search! (possibly setup notifications as well)

The point being, the setup even for an enterprise should not take long at all. All of the components described are often already approved / have deployment patterns within an enterprise (i.e. easy to setup) and the system ships with the required models, setup scripts, etc.

The only “difficult part” is setting up the API calls. However, even the API to build the complex search capabilities, is really quite simple. Only four fields are required.

An example Curl command to Metacortex:

Will get a response like the following:

As you can see there are a few additional fields added, but essentially you get a sentiment, score, and key_word response from the NLP engine, which could be used.

The current v1 API documentation (single comment):

Multiple comments, are simply a list of these objects, contained in the form [{object}, {object}, …, {object}].

That’s really it! The example for instance, was literally just created from hitting an ingestion endpoint with Hacker News comments.


It’s always important to note performance in my book. Today, the system runs as a flask app, where you can lunch multiple flask apps behind a load balancer. It only requires 187Mb of RAM and with a 2.2GHz processor can process ~70 comments / second.

This means, if you had 1 billion comments a year, perfectly distributed in time, the system would be able to handle it on a flask app (1,000,000,000 / (365 * 24 * 60 * 60) = 31.7).

Regarding search, the vast majority of the search is performed on the database, meaning it’s really the database that is the bottleneck. Today with a db.m4.xlarge on AWS, you can within 50ms search for experts and 200 ms for content. The database contains roughly 500,000 authors, 25 million comments, on 125,000 stories, with 10 million unique topics having been mentioned / discussed.

Announcing Metacortex

As some of you may have noticed, recently I took a brief haitus (from blog posting & updates).

There has been a lot going on and although I have been able to manage most technical issues, new features and blog posts have taken a back seat.

Primarily, this is due to my limited bandwidth: 20 – 30 hours a week at the moment. However, I’m excited to announce, we do have some updates this month!


We are pleased to announced!

The goal of which is Enterprise Knowledge Management – find experts, search content, track mood, reduce duplication and more!

The end goal for was always to accomplish two goals:

  1. Debug the search and tracking algorithms for
  2. Keep the lights on while doing #1

To that end, it’s been a smashing success. The algorithm (ExpertRank) is currently being drafted for patenting (provisional filed), and should be filed within the next few weeks. In addition, we have several working demos and I personally (and others) continue to use regularly to make gains on the market (over the last year: 55%, lower than last year). was always intended to be limited in scope; with search being the real end goal. Since 2013, when I wrote the first version of ProjectPiglet, I knew it could be more – a better search engine. This has been exhausted since I’ve started working corporate jobs; I’ve grown to yearn for that search engine.

Even alluding to the fact I didn’t think search was solved (or could be solved) on my personal blog. The point is, if I focused entirely on I would be doing a disservice to myself, my company, and probably to you.

The Future

Unfortunately, this means two things:

  1. We’ll be competing in the enterprise search space, doing B2B
  2. Minimal effort toward new features on

Competing in the enterprise search arena is scary, not only are there big players, there are free players! With that in mind, I have spent the past few months patenting what I could and obfuscating the rest.

As for development on; the demos still rely on the data collected. Meaning, it’s not going anywhere any time soon. It’s the testing ground for the enterprise search. Meaning, it’s necessary to maintain the project, if is to be a success. On the other hand, development outside the scope of enterprise search will be limited (i.e. probably no nicer financial charts). In addition, today I still use regularly to make money.

In other words, will continue – albeit primarily for bug squashing and any new features will be focused towards the end goal of improving search and the algorithms.

Building a Startup – April 2018 Stats

In the last 30 days, I’ve attempted to not change any of the notification settings, nor did I advertise the business. Might sound strange, but I wanted to ensure a stellar experience and you only have one chance at a first impression. You can’t make a good impression with a load of bugs. The goal was two fold, improve for advertising this month (April) and identify attrition rates (if any) and attempt to reduce the attribtion.

Bug Smashing & Feature Improvements

Regarding bugs, this month I’ve closed out 41 issues with the website, mostly identified by current users (hence the ~$2000 from feedback refunds).

The web site repository recap:

The data science engine repository recap:

Combine this with the feedback and I still have 27 outstanding tasks related to, so improvements will continue!

Email Notification Stats

Email notifications are a corner stone of the business. The number of emails we send, along with the open rate, and click rate is a key metric for determining growth / engagement. The fact is, we are already blowing away many of the industry metrics for news notifications. This could be huge for us, in terms of selling the service to other applciations such as Robin Hood (their news section leaves much to be desired).

With that in mind, we’re excited to see we’ve sent just over 2500 emails between March 1, 2018 and March 30, 2018 (a 66% increase):

Our average open rate of 34% (10% decrease) and 5% (40% decrease) click rate:

In contrast, from January 30, 2018 to Feburary 28, 2018 we saw only 1500 emails sent:

With an average open rate of 40% and click rate of 8%:

Unfortunately, that does mean a decrease in engagement across our users (although not massive). Given the tweaks to the notification emails, where there is more detailed information on the news updates, I did suspect the click rate would drop (no need to click a link, if the info is right there).

The more pressing question is the reduced open rate. Upon inspection it appears people didn’t open notifications over the weekend(s) last month, where the month prior we did. In part this could be due to the weather improving, sheer luck, or perhaps the new users habits (I really have no idea). Regardless, we’ll be monitoring open rate and attempting to improve where we can.

We also decided to slightly modify the notification email to be more information dense. Hopefully, this improves the value to opening the email.

Growth & Revenue

This month we have:

  • Gained 8 new paying or feedback providing users
  • Saw a 50% increase in survey completion
  • Lowered the monthly price from $50 to $10
  • Added a referral program (90 days free, per referral)
  • Added payment and survey notifications
  • Improved onboarding (shows a 3x engagement)

Overall, not really a “good” month, we are still averaging slightly over one new user per day since launching the “beta #3”, but would like to see it improve. The more impressive part, is that we haven’t advertised at all in the past 6 weeks. Meaning, every user who’s registered is hearing it from a friend. Importantly, we also have a high conversion rate of ~50% to paying customers, with no disputed charges, and all 8 users came through referrals.

During the month of April 2018, the target is around 30 new users.

We lowered the price for the beta from the standard $50 / month to $10 / month to reflect this. The real goal is to get more feedback, not make money (yet). The incentive to provide feedback in leiu of money has been insanely effective. If we charged the customers outright today, we’d net somewhere around $2000 / month, with applied discounts. However, today we’re doing around $500 / month (enough to cover costs).

Finally, we’ve also seen a massive increase in the number of trends being followed:

This is entirely attributed to our new onboarding system, which we launched March 10th. Each new person who registered started with at least 5 topics being followed and automatically jumped into the tutorial. Those users who registered after the onboarding process change(s) appear to engage 4x (on average) more than prior to the onboarding process change(s).

Future Outlook

Given the improvements regarding onboarding, I’m optimistic. Steady progress is being made technically and users feedback is being addressed almost as quick as it’s coming in. There are a few hurdles regarding some entity recognition I’m working on, but overall the platform is scaling well.

We saw a 66% increase in emails sent, while seeing a 15% drop in open rate, and the ~40% drop in click rate can be explained by the increased information in the email itself. Assuming the open rate and click rate stabilize soon (i.e. above 25% for open rate and around 5% for click rate), we’ll have a stellar product, compared to industry standards.

Finally, the new onboarding (with limited test subjects) has seen a 4x improvement in engagement. We are poised to continue to grow. The goal of beta #3 (the current release) is to figure out how to market to and keep users engaged and I think it’s going very well. For reference, the next beta release (#4) is targeted to release in May – specifically targeting platform enhancements.

April 2018 – Updates

Below is a copy of our monthly newsletter for April 2018


Thank you all for participating in Piglet Beta #3! This month I’ve refunded a total of ~$2000 for those who completed the monthly survey. If you’d like to save money, please fill it out!


March Recap

During March we made steady improvements to the site and platform as a whole. The primary focus has been fixing various issues found by all of you and improving onboarding.

Ways to Make Money & Save

Make Money with Referrals – 90 days free, per referral

Save Money with New Plans – We released a new “Beta #3” plan (it’s 80% off)! You can change your current plan to that plan – go to Account -> Edit Subscriptions, then select the plan.

Exciting April Updates:

  • Landing page & Onboarding
    • Improved “regular price” display (was big, tacky red line)
    • Onboarding category selection, select up to five general categories
    • Mimicked Twitter onboarding process
    • Guided tour always available
  • Users & Subscriptions
    • Launched referral program
    • Growing through organic shares (i.e. no ads or marketing)
    • Added “30 days free” button
    • (Admin Only) Added user specific analytics
    • (Admin Only) Features to enable multiple web portals
  • Assets (Crypto & Stocks)
    • Active users add/edit associated brands for stocks
    • Added sorting Crypto by daily trade volume
    • Fixed stock ticker(s) impacting search results
    • Added real-value for volume and stock/crypto price
    • S&P 500 is up-to-date within 2 – 3 hours for stocks
    • Top 150 cryptocurrencies tracking time is 20 minutes
    • All Cryptocurrencies tracking time has a <3 hour delay
    • Trade potential is calculated every 3 hours
    • Fixed NaN dates bug associated with Safari
  • Trends / Notifications
    • Show trade potential on notifications
    • Removed gray box on header of emails
    • Notifications of pending charges, with free option
      • With this change, I will stop reaching out directly
    • Fixed weekly and monthly notification bug (wasn’t sending)
  • Enterprise

Piglet Blog (posts below):

Upcoming (Next 30 Days)

  • Assets (Crypto & Stocks)
    • Add top stories to asset(s) chart
    • Add net promoter to asset(s) chart
    • (Possibly) Hourly tracking for top stocks and cryptocurrencies
    • (Possibly) Minutely tracking for top stocks and cryptocurrencies
  • Trends / Notifications
    • Feedback mechanism for indicating “these stories don’t match subject” – this will improve notifications
  • (Rollover from last update – delayed)
    • Going to host new Email-Only service for curated emails
    • All users will have basic access to (no assets)
    • All servicing should be one through
  • Additional Blog posts
  • Implement User Feedback & Smash Bugs
  • (Possibly) Add feedback directly through website
    • This is going to replace the google form
  • Architectual / design changes to improve website speed

If you have any questions or concerns, please add to survey and / or reach out.

Happy Investing!

Austin, Piglet Team

Refer a Friend using Stripe and Rails in 15 Lines of Code

A referral program on a SaaS product can often be the difference between success and failure. A famous example of this is Dropbox, which aquired four million users in just fifteen months; their referral program likely driving the companies success (which is just about to IPO).

For a one person shop, the trick is to completely automate the program – requiring no work to maintain. That’s what this article covers! In a few short lines of code, we’ll show you how the referral program was desgined / written, utilizing Ruby on Rails and Stripe.

Our Referral Program

First, it’s important to note there are many ways to design a referral program, ours is as follows:

Get 90 days free subscription ($150 value) in 2 minutes or less!

Here’s how it works:

  1. Refer someone you know providing the following link:
    <link on refer page for>
  2. They click on the link
  3. They sign up, with their credit card and activate their account.
  4. You immediately receive 90 days free on your subscription (in form of “trial”)

That’s it! Round trip, it should take less than two minutes.

Pretty straight forward and we make it worth the users effort! Now, lets look at implementation.

Rails App Design

Every web application is different, however Ruby on Rails forces a particular way to develop, which in this case is a good thing!

For instance, I can assume you have a “user scaffold”, which also comes with a user model & controller, with that model & contoller we can also make the assumption there is a “create function” in said controller. From there, all you need is stripe setup! For the sake of this post, we have a service wrapper around the stripe on – so basically we can treat stripe as a model. It’s really nice, and you can see that in our github gist.

That’s it! That’s all the prerequisites we assume – which is essentially: you’re using Rails and using Stripe!

User Registration Workflow

When a new users comes to register you typically want a few pieces of information. For, we try to keep that to a simple four pieces of information: Name, Email, Password; then you enter your credit card (which we validate). For us that looks like the following:

The question then becomes, how would you want to handle referrals? In our case, it was decided a link will be used for friend referrals and a coupon will be tied back for our “influencer referrals” (i.e. users who get enough people to register, they a cut of the subscription profits).The influencer referral program is not covered in this post.

The link we are looking for should be something like the following:<identifier>

Users then simply have to share that link to get the referral(s), the new potential users will be sent to the signup page and the “referred_by” parameter will then be used by rails to link bake to the referrer. In our case, it’ll just be the user id.

Refer a Friend Implementation (Rails & Stripe)

With the design being in place, it’s really as simple as adding a little bit of code to the create function in user_controller.rb class (14 lines of functional code). I personally add it after the credit card and email validation checks:

Note, our payment system service is in this github gist – it’s merely a wrapper around Stripe.

Then, for some Rails magic, we add the foreign key reference to your user.rb model.

This lets your use it as a reference to link back to a particular user (1 line of functional code):

With both of those in place, you’re done! We utilize the stripe “trial” feature to set the date to 90 days further out. This makes it completely automated dead simple to maintain. The only difficult part (if you’d call it that), was ensuring that the trial times are added to the current trial end date (basically the trial time stacks). Without that, the end date would always be just 90 days from the date of code execution (which would upset customers).

Finally, if you’d like to display who referred who, you can use the following partial:

Again, dead simple – in part because we created that referred_by foreign key reference.

Unfortunately, there are some caveats, although they are minimal. For instance, has users add their credit card at signup. That means, I know with a high probability it’s not a fake user singing up (as we do card validation). This makes it much easier for me to manage any potential users trying to game the system, if your web application doesn’t have that, additional validations should occur.

There’s also the case of shutting down the new account. If a user shuts down account, do you take away the free trial? That’s a question for you, personally I’ve gone both ways.

Regardless, of the caveats above, I hope this helps!

If you’re interested in using, use the coupon code: pigletblog2018

It’s 25% off for 6 months! Plus, clearly we have a refer a friend program!