Metacortex Ingestion API

With the recent announcement of Metacortex, I felt it prudent to share some of the documentation.

The coolest part about the system is that there are only three steps:

  1. Deploy Ingestion Server, Search Server, PostgreSQL backend
  2. Send comments to ingestion endpoint
  3. Search! (possibly setup notifications as well)

The point being, the setup even for an enterprise should not take long at all. All of the components described are often already approved / have deployment patterns within an enterprise (i.e. easy to setup) and the system ships with the required models, setup scripts, etc.

The only “difficult part” is setting up the API calls. However, even the API to build the complex search capabilities, is really quite simple. Only four fields are required.

An example Curl command to Metacortex:

Will get a response like the following:

As you can see there are a few additional fields added, but essentially you get a sentiment, score, and key_word response from the NLP engine, which could be used.

The current v1 API documentation (single comment):

Multiple comments, are simply a list of these objects, contained in the form [{object}, {object}, …, {object}].

That’s really it! The example for instance, was literally just created from hitting an ingestion endpoint with Hacker News comments.


It’s always important to note performance in my book. Today, the system runs as a flask app, where you can lunch multiple flask apps behind a load balancer. It only requires 187Mb of RAM and with a 2.2GHz processor can process ~70 comments / second.

This means, if you had 1 billion comments a year, perfectly distributed in time, the system would be able to handle it on a flask app (1,000,000,000 / (365 * 24 * 60 * 60) = 31.7).

Regarding search, the vast majority of the search is performed on the database, meaning it’s really the database that is the bottleneck. Today with a db.m4.xlarge on AWS, you can within 50ms search for experts and 200 ms for content. The database contains roughly 500,000 authors, 25 million comments, on 125,000 stories, with 10 million unique topics having been mentioned / discussed.

Announcing Metacortex

As some of you may have noticed, recently I took a brief haitus (from blog posting & updates).

There has been a lot going on and although I have been able to manage most technical issues, new features and blog posts have taken a back seat.

Primarily, this is due to my limited bandwidth: 20 – 30 hours a week at the moment. However, I’m excited to announce, we do have some updates this month!


We are pleased to announced!

The goal of which is Enterprise Knowledge Management – find experts, search content, track mood, reduce duplication and more!

The end goal for was always to accomplish two goals:

  1. Debug the search and tracking algorithms for
  2. Keep the lights on while doing #1

To that end, it’s been a smashing success. The algorithm (ExpertRank) is currently being drafted for patenting (provisional filed), and should be filed within the next few weeks. In addition, we have several working demos and I personally (and others) continue to use regularly to make gains on the market (over the last year: 55%, lower than last year). was always intended to be limited in scope; with search being the real end goal. Since 2013, when I wrote the first version of ProjectPiglet, I knew it could be more – a better search engine. This has been exhausted since I’ve started working corporate jobs; I’ve grown to yearn for that search engine.

Even alluding to the fact I didn’t think search was solved (or could be solved) on my personal blog. The point is, if I focused entirely on I would be doing a disservice to myself, my company, and probably to you.

The Future

Unfortunately, this means two things:

  1. We’ll be competing in the enterprise search space, doing B2B
  2. Minimal effort toward new features on

Competing in the enterprise search arena is scary, not only are there big players, there are free players! With that in mind, I have spent the past few months patenting what I could and obfuscating the rest.

As for development on; the demos still rely on the data collected. Meaning, it’s not going anywhere any time soon. It’s the testing ground for the enterprise search. Meaning, it’s necessary to maintain the project, if is to be a success. On the other hand, development outside the scope of enterprise search will be limited (i.e. probably no nicer financial charts). In addition, today I still use regularly to make money.

In other words, will continue – albeit primarily for bug squashing and any new features will be focused towards the end goal of improving search and the algorithms.

New Onboarding Process

In addition to the exciting news about our Refer a Friend Program, we also improved our onboarding.

In this particular case, we’ve stolen a page from the Twitter onboarding process… We noticed we had a 100% convergence rate of customers (from trial to paying), if they were following at least five topics. It appears Twitter made a similar discovery!

As of today, all new users are greated by a selection of topics may want to follow:

These are curated list of topics I personally found to be fairly interesting and representitive as a whole of the category. This should be improved over time, but for the moment each one of those categories follow at least five topics, with some (such as Sicence) following as many as twelve. Users can select as many categories as they’d like.

Further, after that, now all users will be immediately thrown into a tour. Personally, I tend to dislike that option. However, because we are now encouraging the following of topics immediately, the tour link will be less bold and in your face.

Users will only see the large “Click to Begin Guided Tour” if they are not alrady following five topics (aka skipped selection, which we do allow):

If people are interested in following those general categories, you can go to and select them (even if you already have been using the service).

Refer a Friend Program!

Today we’re happy to announce we’ve officially added our referral program!

The details are below!

Refer a Friend

Get 90 days free subscription ($150 value) in 2 minutes or less!

Here’s how it works:

    1. Refer someone you know providing the following link:<link on refer page for>
    2. They click on the link
    3. They sign up, with their credit card and activate their account.
    4. You immediately receive 90 days free on your subscription (in form of “trial”)

That’s it! Round trip, it should take less than two minutes.

Refer Lots of Friends!

In addition, we also have an Influencer Referral Program, accessible to users who have referred over 10 users. We offer a percentage of profits! This works by providing a coupon code and all users who use that coupon code are associated with you.

We provide a 10% – 50% cut of sales (depending on influence).

If intersted, contact us directly.

** Note: We reserve the right to cancel this program at any time.

New Beta Landing Page

A mere four days ago I released the new “Plan” – a cheaper version of Piglet, specifically targeting only emails. Upon further consideration, I’ve decided to remove them from the home page (you can still change the plan via the subscriptions setting). The reasoning behind this is simple – I realized I wasn’t being laser focused on the single product: the financial system.

Why the change?

I realize this was a quick change – you may be asking, what prompted it? The answer is simple and twofold…

First, a user on Reddit suggested I was doing to much on a single landing page. It may sound odd, I’m listening to some random persons advice, but the fact is they’re right‘s landing page was trying to do too much. I have done a lot of research, built more landing pages than I can count, even give advice to others. Apparently, I needed a stranger pointing it out for me to fix it this time.

The second, and perhaps more important reason was pointed out by Patrick McKenzie (patio11 on HN) in his blog post: Business of‘s price point is likely too high ($50 / month) for low touch, non-enterprise sales. A more reasonable price would be $5 / month to $20 / month. I feel proud I’ve gotten an average of 2.2 signups per day at $50 / month (for a consumer product, with sub-par landing page), but I feel limiting scope initially can have it’s benefits.

Structure Going Forward

Reverting back to a single plan (for the moment).

The price of during beta is now going to be the following (for monthly pricing):

  • 80% off for the first 250 users ($10/month)
  • 60% off for users 250 – 500 ($20/month)
  • 40% off for users 500 – 750 ($30/month)
  • 20% off for users 750 – 1000 ($40/month)
  • Full price after

This may change, but for the moment this should help me get more users registering. Further, the following will still be in effect:

  • Refund monthly payment after feedback
  • Coupon codes (25% off – pigletblog2018)
  • Our referral program launches in March

With all of this, I hope we can up to an average of 5 – 10+ registrations a day.

At the end of the beta, it is likely all users will be migrated to the full plan. Current users on the old $50 / month plan will receive an email asking if they wish to be migrated to the cheaper plan (in March) for the duration of the beta.

Separating Out Services

As mentioned in a previous blog post, some services are going to be placed into a separate brand. While still accessible from the main site, there will be a separate site, which hopefully will attract different users. will be a consumer site, focusing on investing.

New Look

With all of that said, we have updated the landing page: